2026-08-29 — views
TTPO Achieves Label-Free Math Reasoning via Asymmetric Policy Optimization
Read this because Engineers can improve LLM reasoning capabilities at test time without relying on expensive or unavailable ground-truth labels.
TTPO uses majority-vote pseudo-labels to match supervised performance on competition-level math benchmarks without ground-truth data.
The Challenge of Label-Free Test-Time Training
Recent advancements in post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have significantly accelerated progress in mathematical reasoning for large language models. However, these approaches typically depend on ground-truth labels, which prevents their application in test-time training (TTT) scenarios where such labels are unavailable. A natural alternative is to replace ground truth with majority-vote pseudo-labels. Yet, this method is inherently fragile. If the majority vote is incorrect, it corrupts the teacher signal and misleads every token in the rollout, leading to degraded performance.
Asymmetric Objective and Token-Level Selection
Researchers observed that the failure mode of pseudo-labels is asymmetric. Rollouts that disagree with the pseudo-label are typically wrong, regardless of whether the vote itself is correct. Building on this insight, they proposed Test-Time Policy Optimization (TTPO). This method employs an asymmetric objective that distills agreeing rollouts via OPSD while penalizing disagreeing rollouts using Grouped RL.
To further refine these branches, TTPO implements token-level selection. In the distillation branch, the method down-weights positions that have already converged. In the RL branch, it penalizes only confident errors. This approach ensures that both updates remain well-grounded even when pseudo-label errors are frequent. Additionally, majority-vote routing provides tighter self-supervision as the model improves over time.
Performance and Generalization
TTPO operates without any labels and matches the performance of label-supervised OPSD on five competition-level benchmarks. In test-time training scenarios, the method raises the performance of Qwen3-1.7B from 38.0% to 45.2%. When applied without thinking tokens, TTPO yields improvements ranging from +25.2% to +36.4%. The method also demonstrates strong cross-task generalization, indicating its potential utility beyond specific mathematical reasoning tasks.
Practitioner note
Developers should monitor the stability of majority-vote pseudo-labels during test-time training, as TTPO relies on the asymmetry of error patterns to function effectively. Implementing token-level selection is critical to prevent the model from over-penalizing confident errors or under-weighting converged positions. This approach offers a viable path for enhancing model reasoning capabilities in environments where ground-truth labels are inaccessible.