On-policy Distillation (OPD)

The Seed framework addresses the limitations of sparse rewards in agentic reinforcement learning by extracting reusable skills from completed interaction trajectories. It utilizes on-policy distillation to provide dense, token-level supervision that evolves alongside the policy's capabilities.

The Seed framework addresses the limitations of sparse rewards in agentic reinforcement learning by extracting reusable skills from completed interaction trajectories. It utilizes on-policy distillation to provide dense, token-level supervision that evolves alongside the policy's capabilities.

This technical paper proposes a new distillation paradigm called Direct-OPD to improve weak-to-strong generalization in reasoning models. By distilling only the policy shift induced by reinforcement learning, the method avoids the capacity limitations inherent in traditional teacher-student imitation.

This technical paper proposes a new distillation paradigm called Direct-OPD to improve weak-to-strong generalization in reasoning models. By distilling only the policy shift induced by reinforcement learning, the method avoids the capacity limitations inherent in traditional teacher-student imitation.