DGX Spark deployment notes
Real-world findings from the NVIDIA DGX Spark / GB10 community on local LLM deployment.
2026-06-10
llama.cpp b9555 Ships Native NVFP4 Kernels for Blackwell SM121, Unlocking Full DGX Spark Performance
llama.cpp b9555 ships native NVFP4 GEMM kernels for Blackwell SM121/GB10 — the first build to bypass the FP16-compute fallback, with an estimated 30–40% single-user decode throughput gain on DGX Spark.
2026-06-09
Google Ships QAT Checkpoints for the Entire Gemma 4 Family: Q4_0 Weights at Near-BF16 Quality
On June 5, 2026, Google released QAT checkpoints for every Gemma 4 size. Q4_0 cuts E4B from 15GB to 5GB and text-only E2B to under 1GB, with day-one support across llama.cpp, Ollama, MLX, vLLM and SGLang.
2026-06-08
Why DGX Spark's GB10 reads as 23 tok/s decode but 1,884 tok/s prefill: a bandwidth-budget breakdown
A verified breakdown of vLLM's June 2026 DGX Spark deployment: a 120B NVFP4 MoE model decodes at ~23 tok/s but prefills at ~1,884 tok/s, and the GB10's 273 GB/s memory bandwidth explains the gap.
2026-06-08
llama.cpp Gains Native Video Input: FFmpeg-Subprocess Decoding Lands in the mtmd Stack
On June 8, 2026, llama.cpp merged PR #24269, adding native video input to its multimodal (mtmd) subsystem. Rather than linking FFmpeg, it shells out to an FFmpeg subprocess to decode frames, and a new lazy-bitmap API
2026-06-08
TensorRT-LLM rc17 lands an NVFP4 MoE backend and NVFP4 KV cache for SM121 (DGX Spark)
TensorRT-LLM v1.3.0rc17 (June 2) adds a FlashInfer NVFP4 MoE backend gated for SM120/SM121, enables NVFP4 KV cache in trtllm-gen attention, and fixes a qwen3 SM120/121 hang — consumer-Blackwell wins for DGX Spark's GB10.
2026-06-07
vLLM 0.22 Adds Multi-Tier KV Cache Offloading: GPU to CPU to Disk for Long-Context Local Serving
vLLM 0.22.0 (May 29, 2026) shipped a multi-tier KV cache offloading framework that cascades cached blocks past CPU DRAM down to disk, with a Python filesystem tier and Mooncake disk backend; the June 5 patch added a few
2026-06-07
vLLM's Official DGX Spark Guide: Why a 120B NVFP4 Model Decodes at ~23 tok/s, and What That Teaches About Bandwidth-Bound Local Inference
On June 1, 2026 the vLLM project published an official guide to running vLLM on NVIDIA's DGX Spark (GB10 Grace Blackwell, sm_121, 128 GB unified memory). Serving Nemotron-3-Super-120B-A12B-NVFP4 from the
2026-06-06
Gemma 4 Multi-Token Prediction Lands in llama.cpp: Self-Speculative Decoding Goes Mainstream for Local Inference
llama.cpp merged native Multi-Token Prediction (MTP) speculative decoding in May 2026 (PR #22673), reporting roughly 2.4x faster single-stream generation on Qwen3.6-27B at about 72% draft acceptance, with the draft head
2026-06-04
NVIDIA's June DGX Spark update turns the desktop box into a 4-node cluster
NVIDIA's June 1 DGX Spark release (DGX OS 7.5.0, driver 580.159.03, NCCL 2.30u1) adds a Sync Cluster Assistant that links up to 3 Sparks without a switch — or 4 with one — into a multi-node inference cluster.
2026-05-29
A Triton FP8 bypass squeezes 17% more NVFP4 speed out of the DGX Spark's GB10
A community patch reroutes NVFP4 weights through the GB10's FP8 tensor cores instead of a slow BF16 fallback, lifting Qwen3.6-35B-A3B from 40.8 to 47.6 tok/s on the DGX Spark.
2026-05-24
llama.cpp merges native MTP speculative decoding — ~2.16× single-stream decode on Qwen3.6 for DGX Spark
PR #22673 lands native multi-token-prediction speculative decoding in llama.cpp (build b9180+). On a GB10 DGX Spark, Qwen3.6-27B Q4_K_M jumps 13.1 to 28.3 tok/s single-request — but regresses under concurrency.
2026-05-09
DGX Spark + Mac Studio disaggregated serving — 2.8× speedup on GPT-OSS-120B by splitting prefill from decode
A community pattern pairs DGX Spark for prefill (~1,723 tok/s on GPT-OSS-120B) with Mac Studio M3 Ultra for decode (819 GB/s) to hit 2.8× end-to-end vs single-Spark FP8.
2026-05-09
Litespark ternary-CPU inference (arXiv 2605.06485) — 9.2× TTFT, 52× throughput, ships pip package
Litespark replaces FP matmul with integer add/sub SIMD on ternary {-1,0,+1} weight networks. 9.2× TTFT, 52× throughput, 14× memory reduction. Pip-installable, HF-integrated.
2026-05-09
llama.cpp lands Gemma 4 26B-A4B NVFP4 (b9080) and MiMo-V2.5 attention kernels (b9085)
llama.cpp b9080–b9085 add native Gemma 4 26B-A4B NVFP4 (52 tok/s on Spark, 82 GB free for KV) and MiMo-V2.5 flash-attention tiles for d_kq=192/d_v=128 GQA shapes.
2026-05-09
TensorRT-LLM v1.3.0rc14 — Qwen3.5 NVFP4 weight-loading fix lands, Mamba-hybrid prefix caching enabled
TRT-LLM 1.3.0rc14 (May 7) lands the Qwen3.5 NVFP4 weight_scales fix, Mamba-hybrid prefix caching, NVFP4 weight-update, DFlash one-model spec-dec, and a Spark-named GEMM perf PR.
2026-05-04
Qwen3 MoE on DGX Spark — NVFP4 vs FP8 benchmarks and what actually works
Community-verified numbers for Qwen3.6-35B-A3B and Qwen3.5-122B-A10B on GB10: NVFP4+MTP reaches 55.9 tok/s single-user, 433 tok/s at c=32. Covers the TRITON-only MoE backend gotcha and the MTP+prefix-cache failure mode.
2026-05-03
DGX Spark deployment notes — what the community is actually fighting (2026 Q2)
Six recurring DGX Spark / GB10 deployment pitfalls from the NVIDIA Developer Forums — most are software, not hardware — plus the MoE + NVFP4/MXFP4 consensus.
2026-05-02
llama.cpp NVFP4 and MXFP4 build guide for GB10 (SM121)
Step-by-step build flags for llama.cpp NVFP4/MXFP4 on DGX Spark GB10 (SM121). gpt-oss-120B MXFP4 hits pp2048=1,980 tok/s and tg32=35 tok/s after the PR #22196 merge.
2026-05-01
vLLM vs llama.cpp vs Ollama on DGX Spark — which inference stack to use
Decision guide for inference stacks on GB10: vLLM wins for MoE+concurrency, llama.cpp for MXFP4 prompts and single-user, Ollama for zero-config dev. Includes NVFP4 tok/s comparison.
2026-04-30
LiteLLM + Claude Code on DGX Spark — LAN serving setup and protocol translation
Route Claude Code API calls to a self-hosted Qwen3 model on DGX Spark via LiteLLM proxy. Covers config, model alias mapping, multi-GPU offload, and the latency tradeoffs vs cloud API.