MuonGS Uses Gram-Schmidt Projection to Boost LLM Pretraining Efficiency
Read this because Builders can achieve faster convergence and better final loss by switching to MuonGS, which avoids the exploration limits of strict norm control.
New MuonGS optimizer uses Gram-Schmidt projection to relax weight-norm constraints, improving validation perplexity and training speed in large language models.