1

No custom engine, no forks: just mainline llama.cpp, tuned and correctness-checked. Sharing the config and what actually moved the needle.

Setup

  • Hardware: 1x NVIDIA DGX Spark, GB10, 128GB unified memory (~121GiB usable)
  • Model: DeepSeek-V4-Flash-0731 (284B total / 13B active MoE, native 1M ctx)
  • Quant: Unsloth UD-IQ2_M (dynamic 2-bit)
  • llama.cpp: build 10235 (221f0f635), basically HEAD as of testing

Download

hf download unsloth/DeepSeek-V4-Flash-0731-GGUF \
    --local-dir ~/models/DeepSeek-V4-Flash-0731-GGUF \
    --include "*UD-IQ2_M*"

Also tested the UD-IQ3_XXS quant side by side. Single-stream and low-concurrency numbers were close to IQ2_M, but it hit a sharp, reproducible throughput dip at concurrency=4 (47.6 to 26.3 tok/s, latency doubling) that IQ2_M never showed. Capability scores were identical between the two, so the dip looks like a memory/scheduling issue at that quant size rather than a quality tradeoff. IQ2_M ended up the more reliable choice under load, so that’s what’s below.

The command

~/tools/llama.cpp/build/bin/llama-server \
  -m DeepSeek-V4-Flash-0731-UD-IQ2_M-00001-of-00003.gguf \
  --alias deepseek-v4-flash-0731 \
  --host 0.0.0.0 --port 8000 \
  --n-gpu-layers 999 --flash-attn on \
  --ctx-size 524288 --parallel 4 --cont-batching \
  --batch-size 2048 --ubatch-size 1024 \
  --jinja --threads 10 --threads-batch 10 \
  --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0 \
  --no-mmap --metrics

524288 total ctx ÷ 4 parallel slots = 131072 tokens guaranteed per concurrent request.

Single-stream decode holds flat (19.3 to 19.7) across generations 2,700+ tokens long, no degradation over a run. Prefill lands 200 to 370 tok/s depending on request shape.

Correctness: validated on a 5-category eval harness (code_generation, code_quality, reasoning, tool_calling, tool_chaining): 96% overall, matching a clean single-stream baseline at full 524288-ctx/4-parallel.

| model             |            test |           t/s |     peak t/s |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |                                                            
|:------------------|----------------:|--------------:|-------------:|-----------------:|-----------------:|-----------------:|                                                            
| deepseek-v4-flash |          pp2048 | 459.40 ± 1.75 |              |  4542.47 ± 17.01 |  4458.10 ± 17.01 |  4542.47 ± 17.01 |                                                            
| deepseek-v4-flash |           tg128 |  19.09 ± 0.02 | 20.00 ± 0.00 |                  |                  |                  |                                                            
| deepseek-v4-flash |          pp2048 | 462.41 ± 0.79 |              |   4513.35 ± 7.55 |   4428.98 ± 7.55 |   4513.35 ± 7.55 |                                                            
| deepseek-v4-flash |           tg512 |  19.17 ± 0.04 | 20.00 ± 0.00 |                  |                  |                  |                                                            
| deepseek-v4-flash |  pp2048 @ d4096 | 449.21 ± 1.96 |              | 13762.10 ± 59.81 | 13677.73 ± 59.81 | 13762.10 ± 59.81 |                                                            
| deepseek-v4-flash |   tg128 @ d4096 |  18.90 ± 0.06 | 19.00 ± 0.00 |                  |                  |                  |                                                            
| deepseek-v4-flash |  pp2048 @ d4096 | 448.69 ± 1.52 |              | 13777.85 ± 46.17 | 13693.48 ± 46.17 | 13777.85 ± 46.17 |                                                            
| deepseek-v4-flash |   tg512 @ d4096 |  18.95 ± 0.02 | 19.00 ± 0.00 |                  |                  |                  |                                                            
| deepseek-v4-flash | pp2048 @ d16384 | 419.14 ± 0.61 |              | 44060.04 ± 64.12 | 43975.67 ± 64.12 | 44060.04 ± 64.12 |                                                            
| deepseek-v4-flash |  tg128 @ d16384 |  18.15 ± 0.00 | 19.00 ± 0.00 |                  |                  |                  |                                                            
| deepseek-v4-flash | pp2048 @ d16384 | 418.91 ± 0.70 |              | 44084.43 ± 72.98 | 44000.07 ± 72.98 | 44084.43 ± 72.98 |                                                            
| deepseek-v4-flash |  tg512 @ d16384 |  18.23 ± 0.01 | 19.00 ± 0.00 |                  |                  |                  |

Decode barely moves out to 16K of prior context (19.09 → 18.15 tok/s), prefill drops about 9%. Combining depth with concurrency is a different story: 4 concurrent requests each carrying 16K of prior context collapsed to 6.3 tok/s combined (not per-request) in our concurrency sweep, with per-request throughput wildly uneven (5.1 ± 3.6 tok/s) rather than evenly split. Long single-agent context holds up fine here; several long-context agents at once does not, at least not without more headroom than a single Spark gives you.

2

Yep… it is possible to get fairly good performance out of llama.cpp here. You could add DSpark to get a significant ~70% output speed boost.

The main problem is that llama.cpp can’t represent the DSv4 KV cache in its native format, which is a mixture of mostly fp8 and fp4, so it takes a lot more memory in llama.cpp at 16-bit, or llama.cpp hurts the quality if you store it in 8-bit. This, combined with getting double the prompt processing throughput, is what got me to try out GitHub - Entrpi/ds4-on-spark: Entrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream prefill, ~1.5x decode, DSpark, and full continuous batch support · GitHub which has been mentioned on this forum. I can fit over 1M context in about 107GB of memory. For this model, llama.cpp also doesn’t support any unified KV cache, so you are forced to split what limited context you have over multiple slots if you want to run things in parallel, whereas DS4 can divide the total KV cache more flexibly.

I think llama.cpp will eventually have these things too, and then I’d probably prefer to switch back, but it’s nice to explore the options.

3

Yeah, tried Entrpi’s ds4-on-spark too. Got the prefill boost, but DSpark itself never fired for me, every request hit 0.0% accept because the continuous-batch path got rejected on the memory floor at my target context.

Single-stream ended up basically the same as llama.cpp (~22 vs ~19.7 tok/s), not the 70% bump.

Also couldn’t get anywhere near 107GB for 1M context. I was at ~117GB used with no drafter and just 131K. Would genuinely like to see your exact command/quant if you’re willing to share.

KV cache point holds up though, q8_0 gave me garbled output on this model too.

4

There are some long context settings that help. I don’t remember which ones were most impactful.

This is roughly the command I am using:

DS4_CUDA_NO_HBM_CACHE=1
DS4_BATCH_FIT_HEADROOM_MB=6272
DS4_BATCH_VMM_BUDGET_MB=6144
DS4_SERVER_COALESCE_MAX=8
DS4_CONT_PREFILL_CHUNK=2048
DS4_CONT_CAPTURE=1
DS4_SERVER_DEFAULT_TEMP=0
./ds4-server --cuda \
  -m ~/models/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf \
  --dspark ~/models/DSpark-drafter-Q2K-Q8-0731.gguf \
  -c 1048576 \
  --kv-disk-dir ~/.cache/ds4-kv \
  --kv-disk-space-mb 32768 \
  --host 0.0.0.0 --port 8000