FSPO: A new optimization method for reinforcement learning in language models
In brief
- FSPO uses historical token distributions as fixed temporal targets to avoid instability from evolving teacher models.
- The method improves average accuracy by 4 points on reasoning and tool-use benchmarks.
- FSPO reduces peak memory from 70 GB to 56 GB on Olmo3-7B-Instruct within a 5-hour training window.
Researchers introduce Function-Space Policy Optimization (FSPO), an optimization method for reinforcement learning in language models. FSPO leverages historical token distributions as fixed temporal targets, eliminating the need for an evolving teacher model. It also incorporates privileged disagreement correction and adaptive temporal memory to enhance the reliability of demonstrations.
Results show an average improvement of 4 percentage points on reasoning and tool-use benchmarks, with a memory reduction from 70 GB to 56 GB on Olmo3-7B-Instruct under a 5-hour training budget.
This summary gives you the essentials. The full article, its figures and quotes are on Hugging Face Daily Papers.
Read the original articleAI-generated summary of the source, checked by rules. How we write · Report an error