What On-Policy Distillation Quietly Drops — and When It Matters
Future credit, local normalization, and when these two omissions change optimization—or the target itself.
A close reading of recent on-policy distillation work reveals two commonly omitted terms. One changes the optimization path; the other can change the fixed point for extrapolated and token-level multi-teacher targets.