2026-08-25 👁️ — views

New Method Makes Compressed 4-Bit Models Beat Full-Precision Versions

Read this because Builders should care because this method recovers capabilities usually lost during compression and quantization.

A new healing technique allows a compressed 4-bit model to outperform its full-precision original on seven benchmarks.

The Problem With Standard Compression

Making a large language model smaller almost always comes with a cost. The standard recipe involves compressing the architecture by removing layers, heads, or neurons, then quantizing weights down to 4 bits. While both steps save resources, they systematically degrade capabilities like reasoning and code generation. Consequently, deployment pipelines add a recovery step called healing before production.

Why Existing Methods Fail

Most efficiency pipelines rely on quantization-aware training (QAT) or quantization-aware distillation (QAD). QAT re-runs expensive post-training processes through a noisy forward pass, which can be unstable. QAD avoids this by distilling from a frozen full-precision teacher. However, once a model undergoes structural compression, the original full-precision version no longer exists as an exact match. The only available teacher is a recovered checkpoint, which acts as a degraded target and caps the student’s accuracy.

Introducing Quantization-Aware Healing

The new approach, Quantization-Aware Healing (QAH), removes this ceiling by distilling directly from the original, pre-compression model rather than the recovered one. The teacher is full-size and full-precision, while the student is half the size running in MXFP4. Because the teacher’s output distribution is architecture-agnostic, the mismatch does not prevent knowledge transfer. Under QAH, quantization becomes a second pass of distillation against the original teacher, providing supervision the bfloat16 checkpoint never received.

Results on Benchmarks

The team applied QAH to a GPT-OSS 120B model compressed to 60B parameters and quantized to MXFP4. The resulting 4-bit model beats its own full-precision version on seven of nine benchmarks. It is smaller, cheaper to run, and more accurate than the checkpoint it was quantized from.

Comparison With QAT

In a head-to-head comparison against QAT using a GPT-OSS 9B model, both methods reached similar peak accuracy. However, QAH reached its peak in about 100 steps, roughly seven times faster than QAT’s 700 steps. Furthermore, QAH stays stable after the peak, whereas QAT collapses sharply once past its best point.

Practitioner note

Developers should consider using this distillation strategy to avoid the instability risks associated with cross-entropy objectives that push models indefinitely toward hard labels. By anchoring the student to a fixed teacher distribution, teams can safely serve checkpoints without worrying about degradation over time.

Sources

• Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original ↗

Tags