The paper introduces Direct On-Policy Distillation (Direct-OPD) as a method to transfer reasoning capabilities from large models to smaller models by focusing on policy shifts rather than absolute policies.
TL;DR
This technical paper proposes a new distillation paradigm called Direct-OPD to improve weak-to-strong generalization in reasoning models. By distilling only the policy shift induced by reinforcement learning, the method avoids the capacity limitations inherent in traditional teacher-student imitation.
The research addresses a critical bottleneck in the development of frontier reasoning models: how to effectively transfer high-level reasoning capabilities from large 'teacher' models to smaller, more efficient 'student' models. Current methodologies rely heavily on On-Policy Distillation (OPD), where students are trained on states sampled under teacher supervision. However, existing OPD methods and their hybrids often suffer from a capacity ceiling; if the teacher model is used to imitate an absolute policy, the student effectively inherits the limitations of that teacher. Furthermore, research indicates that small models struggle significantly when attempting to copy much stronger reasoners directly. To solve this, the paper introduces Direct On-Policy Distillation (Direct-OPD). Unlike standard approaches that attempt to mimic the teacher's final output distribution, Direct-OPD focuses exclusively on the policy shift—the difference between the post-RL policy and the reference policy (log πT - log πTref). By discarding the absolute policy and focusing only on the improvements introduced by reinforcement learning, the student model can more effectively learn the underlying reasoning logic without being constrained by the teacher's absolute performance bounds. This technique facilitates better weak-to-strong generalization, allowing smaller models to potentially extrapolate beyond the immediate imitation of the teacher's final state.
Reinforcement Learning
On-Policy Distillation (OPD)
Direct On-Policy Distillation (Direct-OPD)
Large Language Models (LLMs)
KL-constrained RL
Read paper →
Paper 2
HuggingFace Papers
The article introduces KVpop, a novel method for efficient Key-Value (KV) cache compression in large language models using predictive online pruning.
TL;DR
This technical paper presents KVpop, a technique for compressing the KV cache in transformer models via predictive online pruning. It optimizes the computation of attention targets by reusing sparse log-normalizers and utilizing efficient data structures like Fenwick trees.
The research introduces KVpop, an advanced approach to Key-Value (KV) cache compression designed to maintain model performance while significantly reducing memory overhead during inference. The core innovation lies in its ability to predictively prune the KV cache using a student-teacher framework. To avoid the massive computational cost of calculating dense causal attention probabilities, KVpop implements a transposed-attention target computation. This method swaps query and key roles in an efficient attention kernel, such as FlexAttention, to approximate future-attention mass without materializing large S×S matrices. By reusing sparse log-normalizers from the student pass, the system achieves high accuracy with minimal added inference-time overhead. Furthermore, the algorithm manages a top-k sparse attention pattern by maintaining a union of sink tokens, a recent window, and high-priority tokens. To keep this process efficient as the sequence length grows, the authors utilize a Fenwick tree to track token ranks, allowing for a query-specific cutoff computation in O(S log S) time. This ensures that the sparse mask is generated dynamically within fused kernels, optimizing both space and time complexity.
KVpop
FlexAttention
Fenwick tree
Transformer Attention Kernels
Sparse Attention
Read paper →
Paper 3
HuggingFace Papers
The introduction of VLA-Corrector, a lightweight framework designed to improve the robustness and efficiency of Vision-Language-Action models through adaptive action horizon management.
TL;DR
VLA-Corrector is a new inference framework that mitigates the risks of error accumulation in action-chunked robotic policies. By using latent-space monitoring and gradient guidance, it allows robots to dynamically truncate stale actions and replan when environmental deviations are detected.
Vision-Language-Action (VLA) models have become a cornerstone of embodied intelligence, yet they face a fundamental trade-off between computational efficiency and closed-loop reactivity. Most current models utilize 'action chunking' to predict multiple future steps in one pass, reducing the frequency of expensive policy calls. However, this creates an 'open-loop blind spot' where the robot continues executing pre-planned actions even if physical perturbations, such as slippage or collisions, occur. This lack of reactivity leads to compounding errors and task failure. The paper presents VLA-Transformer (VLA-Corrector), a framework that operates without modifying the original model weights. It employs two primary mechanisms: first, a Latent-space Vision Monitor (LVM) that tracks discrepancies between expected and actual visual features to trigger an interrupt; second, Online Gradient Guidance (OGG) which uses detected deviations to inject corrective gradients during re-inference. This approach effectively transforms a static action horizon into an adaptive one. Experimental results on various VLA backbones like π0.5, SmolVLA, and X-VLA show significant improvements in success rates and 'success-per-call' efficiency, particularly in complex, contact-rich manipulation tasks.
Vision-Language-Action (VLA) models
Latent-space Vision Monitor (LVM)
Online Gradient Gradient Guidance (OGG)
Diffusion Models
Flow Matching
π0.5
SmolVLA
X-VLA
Read paper →
The introduction of ReChannel, a method that repurposes text-to-image Diffusion Transformers (DiTs) for dense prediction by reading out task-native pixel-space fields directly instead of generating RGB targets via VAE decoders.
TL;DR
This paper proposes ReChannel, a novel architecture that transforms text-to-image models from RGB generators into efficient dense prediction engines. By treating transformer tokens as spatial carriers for task-specific data rather than RGB pixels, the method achieves new state-of-the-art performance with much higher computational efficiency.
The research addresses a fundamental inefficiency in using large-scale text-to-image models for dense prediction tasks like depth estimation, segmentation, and matting. Current approaches typically treat these tasks as an image-to-image translation problem, encoding task targets into an RGB-trained VAE latent space and decoding them back to pixels. The authors argue that this 'generation' interface is unnecessary because dense prediction requires pixel-accurate task fields, not the reconstruction of complex RGB textures. They propose 'ReChannel,' which leverages the inherent patch-based spatial structure of Diffusion Transformers (DiTs). In ReChannel, the pretrained DiT acts as a field organizer; the input passes through a standard VAE encoder, but the output bypasses the decoder entirely. Instead, a lightweight token-local linear head maps adapted tokens directly to task-native pixel patches. This approach was validated using the FLUX-Klein backbone across six different dense prediction tasks and over twelve benchmarks. The results show that ReChannel not only reaches state-of-the-art performance in areas like KITTI depth and trimap-free matting but also provides a massive speedup, performing up to 2.48x faster than previous generative editing methods while using significantly fewer parameters for the output head.
ReChannel
Diffusion Transformer (DiT)
FLUX-Klein
VAE (Variational Autoencoder)
LoRA (Low-Rank Adaptation)
DINOv2
SAM (Segment Anything Model)
Read paper →
Paper 5
HuggingFace Papers
The research introduces BRAID, a method using Reinforcement Learning (RL) to enhance the multi-modal reasoning and visual perception capabilities of unified multimodal models.
TL;DR
This paper presents BRAID, a framework that bridges interleaved multi-modal reasoning by treating it as a unified decision process via Reinforcement Learning. The method demonstrates that RL can close the performance gap between generative unified models and dedicated understanding-only VLMs.
The research addresses the historical deficit in visual reasoning capabilities within unified multimodal architectures, which often struggle to match the performance of understanding-only models. By implementing a training pipeline that starts with Supervised Fine-Tuning (SFT) to establish a reliable baseline and follows with Reinforcement Learning (RL), the authors introduce BRAID. This method demonstrates significant improvements in spatial reasoning tasks like SAT and visual perception benchmarks like V*Bench, even surpassing GPT-4o on specific metrics using only 7B parameters. The study evaluates several data allocation regimes and rollout-to-update ratios, concluding that a 2:1 ratio provides the necessary stability for gradient estimation and prevents policy oscillation. Furthermore, the researchers prove through Maj@nn and Avg@nn metrics that RL does not merely increase output diversity but actively enhances the model's underlying reasoning capacity and solution space exploration.
BRAID
GPT-4o
Qwen2.5-VL
BAGEL
Janus-Pro
Chameleon
VLMEvalKit
DiffusionNFT
Read paper →