On-Policy Distillation (OPD)

The researchers present Relay-OPD, a method designed to fix the issue of 'prefix failure' in large language model distillation. By allowing a teacher model to briefly intervene when it detects a reasoning deviation, the system improves student accuracy on mathematical benchmarks with minimal computational overhead.

The Seed framework addresses the limitations of sparse rewards in agentic reinforcement learning by extracting reusable skills from completed interaction trajectories. It utilizes on-policy distillation to provide dense, token-level supervision that evolves alongside the policy's capabilities.

This technical paper proposes a new distillation paradigm called Direct-OPD to improve weak-to-strong generalization in reasoning models. By distilling only the policy shift induced by reinforcement learning, the method avoids the capacity limitations inherent in traditional teacher-student imitation.