Paper 1
Weak-to-Strong Generalization via Direct On-Policy Distillation
The paper introduces Direct On-Policy Distillation (Direct-OPD) as a method to transfer reasoning capabilities from large models to smaller models by focusing on policy shifts rather than absolute policies.
TL;DR
This technical paper proposes a new distillation paradigm called Direct-OPD to improve weak-to-strong generalization in reasoning models. By distilling only the policy shift induced by reinforcement learning, the method avoids the capacity limitations inherent in traditional teacher-student imitation.
The research addresses a critical bottleneck in the development of frontier reasoning models: how to effectively transfer high-level reasoning capabilities from large 'teacher' models to smaller, more efficient 'student' models. Current methodologies rely heavily on On-Policy Distillation (OPD), where students are trained on states sampled under teacher supervision. However, existing OPD methods and their hybrids often suffer from a capacity ceiling; if the teacher model is used to imitate an absolute policy, the student effectively inherits the limitations of that teacher. Furthermore, research indicates that small models struggle significantly when attempting to copy much stronger reasoners directly. To solve this, the paper introduces Direct On-Policy Distillation (Direct-OPD). Unlike standard approaches that attempt to mimic the teacher's final output distribution, Direct-OPD focuses exclusively on the policy shift—the difference between the post-RL policy and the reference policy (log πT - log πTref). By discarding the absolute policy and focusing only on the improvements introduced by reinforcement learning, the student model can more effectively learn the underlying reasoning logic without being constrained by the teacher's absolute performance bounds. This technique facilitates better weak-to-strong generalization, allowing smaller models to potentially extrapolate beyond the immediate imitation of the teacher's final state.
Paper 2
KVpop -- Key-Value Cache Compression with Predictive Online Pruning
The article introduces KVpop, a novel method for efficient Key-Value (KV) cache compression in large language models using predictive online pruning.
TL;DR
This technical paper presents KVpop, a technique for compressing the KV cache in transformer models via predictive online pruning. It optimizes the computation of attention targets by reusing sparse log-normalizers and utilizing efficient data structures like Fenwick trees.
The research introduces KVpop, an advanced approach to Key-Value (KV) cache compression designed to maintain model performance while significantly reducing memory overhead during inference. The core innovation lies in its ability to predictively prune the KV cache using a student-teacher framework. To avoid the massive computational cost of calculating dense causal attention probabilities, KVpop implements a transposed-attention target computation. This method swaps query and key roles in an efficient attention kernel, such as FlexAttention, to approximate future-attention mass without materializing large S×S matrices. By reusing sparse log-normalizers from the student pass, the system achieves high accuracy with minimal added inference-time overhead. Furthermore, the algorithm manages a top-k sparse attention pattern by maintaining a union of sink tokens, a recent window, and high-priority tokens. To keep this process efficient as the sequence length grows, the authors utilize a Fenwick tree to track token ranks, allowing for a query-specific cutoff computation in O(S log S) time. This ensures that the sparse mask is generated dynamically within fused kernels, optimizing both space and time complexity.
KVpop
FlexAttention
Fenwick tree
Transformer Attention Kernels
Sparse Attention
Read paper →
Paper 3
From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models
The introduction of ReChannel, a method that repurposes text-to-image Diffusion Transformers (DiTs) for dense prediction by reading out task-native pixel-space fields directly instead of generating RGB targets via VAE decoders.
TL;DR
This paper proposes ReChannel, a novel architecture that transforms text-to-image models from RGB generators into efficient dense prediction engines. By treating transformer tokens as spatial carriers for task-specific data rather than RGB pixels, the method achieves new state-of-the-art performance with much higher computational efficiency.
The research addresses a fundamental inefficiency in using large-scale text-to-image models for dense prediction tasks like depth estimation, segmentation, and matting. Current approaches typically treat these tasks as an image-to-image translation problem, encoding task targets into an RGB-trained VAE latent space and decoding them back to pixels. The authors argue that this 'generation' interface is unnecessary because dense prediction requires pixel-accurate task fields, not the reconstruction of complex RGB textures. They propose 'ReChannel,' which leverages the inherent patch-based spatial structure of Diffusion Transformers (DiTs). In ReChannel, the pretrained DiT acts as a field organizer; the input passes through a standard VAE encoder, but the output bypasses the decoder entirely. Instead, a lightweight token-local linear head maps adapted tokens directly to task-native pixel patches. This approach was validated using the FLUX-Klein backbone across six different dense prediction tasks and over twelve benchmarks. The results show that ReChannel not only reaches state-of-the-art performance in areas like KITTI depth and trimap-free matting but also provides a massive speedup, performing up to 2.48x faster than previous generative editing methods while using significantly fewer parameters for the output head.
Paper 5
DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
The article introduces DSpark, a novel confidence-scheduled speculative decoding method that utilizes semi-autoregressive generation to accelerate large language model inference.
TL;DR
This technical paper presents DSpark, a new approach to speculative decoding designed to enhance the speed of autoregressive generation. It contextualizes the work within the broader landscape of drafting architectures and parallel generation strategies.
The article explores the evolution of speculative decoding, a technique used to accelerate the inference of large language models by separating the proposal of tokens from their verification against a target model. The efficiency of this process is highly dependent on the accuracy and speed of the 'drafter.' Historically, researchers have utilized small standalone language models as drafters, but recent innovations have integrated multi-token heads or feature extrapolators directly into the target model's architecture to improve performance. Other specialized strategies mentioned include self-speculation through early exits, dynamic vocabulary compression, prompt lookup, and retrieval-based methods. To address the sequential bottleneck inherent in traditional drafting, newer research has moved toward parallel or blockwise generation techniques such as Medusa, P-EAGLE, PARD, DART, and DFlash. Furthermore, some advanced methods like DDTree, TAPS, and JetSpec expand the draft chain into verifiable trees to maximize throughput. The paper positions DSpark within this landscape, highlighting its use of confidence-scheduled semi-autoregressive generation as a way to optimize the balance between drafting speed and verification accuracy.
DSpark
Medusa
P-EAGLE
PARD
DART
DFlash
DDTree
TAPS
JetSpec
Domino
DFlare
Read paper →