2026-05-09 — 회 조회
Litespark 삼진 CPU 추론(arXiv 2605.06485) — 9.2× TTFT, 52× 처리량, pip 패키지 제공
Litespark는 삼진 {-1,0,+1} 가중치 네트워크에서 FP 행렬곱을 정수 가감 SIMD로 대체. 9.2× TTFT, 52× 처리량, 14× 메모리 감소. Pip 설치, HF 통합.
번역 주: 본 글의 핵심은 기술 참조(코드, URL, 성능 수치, 벤치마크)입니다. 명령어·식별자의 정확성을 유지하기 위해 본문은 영어 그대로 두고, 제목과 요약만 번역했습니다. Submitted to arXiv on May 7, 2026, Litespark-Inference (Dade, Morri, Rahat, Pal) replaces FP matrix multiplication with integer add/subtract SIMD kernels for ternary 1 weight networks. The headline numbers vs PyTorch baseline on Apple Silicon:
| Metric | Improvement |
|---|---|
| Time-to-first-token | 9.2× faster |
| Throughput | 52× higher |
| Memory footprint | 14× smaller |
Comparable gains on Intel and AMD x86. The crucial detail: it ships as a pip-installable package that integrates with HuggingFace Transformers — so it’s not just a paper, it’s a working tool you can pip install today.
Why ternary matters for inference
Ternary weights 1 can be encoded in 1.58 bits and computed using only integer add/subtract operations — no multiplication required. This unlocks:
- No FPU dependency — runs on the cheapest ARM cores
- SIMD-friendly — integer add/sub vectorizes trivially on NEON / AVX2 / AVX-512
- Lower energy per token — the dominant arithmetic op (multiply) is replaced by the cheapest one (add)
The catch: full-precision LLMs aren’t natively ternary. Litespark targets ternary-trained networks specifically (BitNet-family models and successors). Using it for a dense Qwen / Llama is a separate pipeline — you need a ternary distillation step.
Why this matters for DGX Spark operators
DGX Spark has a 20-core Grace CPU alongside the GB10 Blackwell GPU. Most operators leave the Grace cores idle during inference. Litespark gives you a credible reason to use them:
-
Draft model for speculative decoding. If you’re running Qwen3.6-35B-A3B on the GPU, a ternary draft model on the Grace cores can produce candidate tokens in parallel, leaving the GPU as the verifier. This is the same pattern as MTP-1, but the draft runs on different silicon — no GPU contention.
-
Routing / classification offload. Small ternary classifiers (intent detection, content moderation, code-vs-prose routing) can run on the Grace side without stealing GPU cycles from your main serving loop.
-
Embedding generation. Ternary embedding models scale near-linearly with CPU cores. 20 Grace cores × Litespark kernels gives respectable throughput for RAG indexing alongside GPU serving.
What to do
pip install litespark-inference
# Try a ternary draft model alongside your Qwen3.6 verifier
python -c "
from litespark import LitesparkLM
draft = LitesparkLM.from_pretrained('bitnet-b1.58-3b')
# Use as draft model for spec-dec against your main Spark Qwen3.6 verifier
"A 30-minute experiment is appropriate scope: measure draft acceptance rate on your actual workload mix. If acceptance hits 70%+, the spec-dec lever is worth wiring into your serving stack. If it’s below 50%, the ternary draft and the FP target are too dissimilar — you’re better off with a homologous draft model.
This pattern also bridges to disaggregated serving: ternary draft on CPU → FP NVFP4 verifier on GPU. The two phases never compete for the same memory bandwidth.