2026-10-28 — views
MuonGS Uses Gram-Schmidt Projection to Boost LLM Pretraining Efficiency
Read this because Builders can achieve faster convergence and better final loss by switching to MuonGS, which avoids the exploration limits of strict norm control.
New MuonGS optimizer uses Gram-Schmidt projection to relax weight-norm constraints, improving validation perplexity and training speed in large language models.
The Limits of Strict Norm Control
Weight decay and explicit weight-norm constraints are known to improve optimizer performance in language model pretraining. However, the theoretical understanding of why this control helps and how much is required has remained incomplete. Researchers analyzed norm control in the Muon optimizer using the accumulated effective learning rate, defined as the sum of the nominal learning rate divided by the Frobenius norm of the weights over the training trajectory.
The analysis reveals that vanilla Muon can accumulate only logarithmically, which potentially limits exploration and explains its slower empirical progress. In contrast, variants like MuonW and MuonH guarantee at least linear accumulation under specific conditions. This distinction highlights that while bounded-norm control is beneficial, it may be too restrictive for optimal exploration.
Introducing MuonGS
To address these limitations, the researchers propose MuonGS, an optimizer that employs Gram-Schmidt (GS) projection. This technique removes the radial component of each Muon update without fixing the weight norm. By allowing sufficiently slow norm growth, MuonGS guarantees faster accumulation than the logarithmic worst case, enabling less restrictive optimizer designs.
Theoretical proofs demonstrate that with bounded update directions and a constant nominal learning rate, the weight norm in MuonGS grows at most proportionally to the square root of the step count. Consequently, the accumulated effective learning rate scales with the square root of the total steps, ensuring sustained accumulation even in later stages of training. For smooth scale-invariant objectives, MuonGS also establishes a stationarity guarantee of O(T^-1/2).
Experimental Results
Pretraining experiments were conducted on 160M- and 400M-parameter dense models, as well as a 135M active- and 1B total-parameter Mixture of Experts (MoE) model. The results indicate that MuonGS achieves lower validation perplexity than MuonH at long training horizons. Notably, for the dense model, MuonGS delivers an estimated token-equivalent speedup of up to approximately 15% at matched validation loss compared to MuonH.
Practitioner note
Developers should consider replacing standard Muon or MuonH with MuonGS when pretraining large models to potentially gain significant speedups at equivalent loss levels. Monitor weight norm growth during training, as MuonGS relies on relaxed constraints that allow norms to grow proportionally to the square root of steps rather than remaining fixed.