DeepSeek-V4.1-Flash at 1M context on four DGX Sparks plus one RTX 5090

Headlines

Every number on this page was measured on this stack on 2026-09-14 (Sydney) unless it is explicitly labelled as published by someone else. Prefill and decode are always reported separately: prefill is prompt tokens divided by time to first token; decode is output tokens per second after the first token, on one stream unless stated.

1 · The architecture

A mixture-of-experts model spends its decode time on two very different jobs. Attention over the context wants a fast GPU with the KV cache close by. The routed experts want a lot of memory, because every token reads a slice of hundreds of gigabytes of expert weights. Attention-FFN disaggregation splits the model along that seam: one GPU owns attention and everything sequential, and a pool of memory-rich nodes owns the experts. Per layer, the coordinator sends a small activation vector out over the fabric and gets the experts' output back.

The engine is DS41RT, whose published reference pairs a 96 GB RTX PRO 6000 coordinator with four Sparks. This deployment ports the coordinator to a 32 GB RTX 5090. The published coordinator image will not start on a 5090: its ahead-of-time kernels are exported for the build GPU's streaming-multiprocessor count (188 on the 96 GB card) and the engine rejects any other. Rebuilding at the published revision with the export targeting the 5090's 170 SMs produced an image that is label-identical to the vendor's apart from that constant.

2 · Uplift over four Sparks alone ★ pending a true like-for-like run

The comparison that matters is the same model on the same four Sparks without the fifth GPU: the tensor-parallel vLLM deployment published by tonyd2wild. The baseline bars below are that author's published boot-10 figures, measured with his prompt set and method; the hybrid bars are ours. The two prompt sets differ, so treat the decode panels as indicative until the same ladder has been run on both stacks.

Read it this way. Single-stream decode is about the same on both, because on both stacks a decode step is bound by streaming expert weights out of the Sparks' unified memory. What the fifth GPU buys is everything that was starved on the Sparks: the prefill runs on a card with far more compute per watt of memory traffic and a prefill lane four times wider, the KV pool is 2.3 times larger at this configuration and seven times larger at the pool-first alternate, because the coordinator's memory is spent on cache rather than expert weights, and concurrent decode has more room to batch.

3 · Prefill

Cold prefill, unique prompts, temperature 0, speculative drafter on. Prefill throughput is prompt tokens divided by time to first token. Each rung is one request.

Prefix retention is what makes 1M usable. With retention off, a repeated 170k-token prompt costs the full cold prefill again (83 s when this was measured at capacity 256). With 24 retained prefixes it returns in 0.3 s, and the engine's memory plan is byte-identical either way: the retention banks live inside the fixed 2 GiB runtime headroom, so retention costs nothing from the KV pool.

Concurrent prefill. Measured on the pool-first configuration: sixteen 105k-token requests released together all completed at 407.6 s wall each, perfectly batched, for about 3,340 tok/s of aggregate prefill against 2,000 solo at that capacity. The single prefill lane serialises admission, so beyond two to four prefill-heavy streams extra concurrency buys queueing rather than throughput. The same shapes have not been re-run at the current default.

For scale: the vendor's 96 GB reference coordinator at prefill capacity 4096 cold-prefills its 1M prompt in 322 s (their figure, about 3,300 tok/s); this card does the 679k-token rung at 4,020 tok/s with a quarter of the prefill lane.

4 · Decode

Decode is output tokens per second after the first token. It depends on the prompt, because the speculative drafter accepts more draft tokens on code and structured text than on prose. Both numbers below are one stream, temperature 0, drafter on.

The engine requires temperature 0 for its native sampling path, so the bench ran with that one sampling change; acceptance length is not reported because DS41RT does not expose vLLM-style speculative-decoding metrics.

5 · Concurrency and the pool

The admission cap is 16 streams at either operating point, and a lane costs 6 MB, so what bounds concurrency is the pool. At the default (2.5M tokens, about 2.4 full contexts) many simultaneous long sessions are pool-bound by construction. The two large shapes below were measured on the pool-first alternate (capacity 256, 7.79M-token pool) and have not been re-run at the default; decode-side concurrency (section 4) is independent of the prefill capacity.

• Sixteen simultaneous 105k-token requests: 16 of 16 served (pool-first alternate), identical wall times, no errors.
• Eight simultaneous 679k-token requests: 8 of 8 served, zero pool refusals (pool-first alternate), each completing at 1,806.6 s. The pool never declined an allocation because the prefill lane admits work serially, so live-token demand stayed under the dynamic ceiling.

The practical limiter is the prefill lane rather than KV memory: at long context the fleet is happiest with a few prefill-heavy streams and many decode-heavy ones. If a workload needs many concurrent long sessions, the pool-first alternate is the right configuration.

6 · Fitting the coordinator into 32 GB

The coordinator's weights are modest at 17.5 GiB. What made the port hard is that the engine's two execution lanes, their arenas and the default KV plan were sized for a 96 GB card. Four settings decided the outcome, each measured before it was adopted.

1. Explicit memory reservation at 97%. The automatic pool planner budgets for the worst case and refuses to start on 32 GB. Reserving explicitly lets the planner size the pool itself (2.5M tokens at capacity 1024, 7.79M at 256) with about 3 GiB of device memory to spare, and the engine reaches readiness in seconds.
2. Two operating points, one flag. The dSpark drafter costs 7.95 GB resident and stays on (it is worth about 1.8× on decode). Prefill capacity 4096 runs out of memory outright on this card, so the choice is between capacity 1024, which leaves 2.07 GiB for the pool (26.35 GiB occupied, 2.5M tokens), and capacity 256, which leaves 7.4 GiB (7.79M tokens). Capacity 1024 is the default: prefill at 170k tokens is 2.69× faster (5,377 vs 2,000 tok/s) and decode is unchanged. Capacity 256 is the pool-first alternate for many concurrent long sessions. The coordinator's batch flag and the ranks' capacity flag must agree, and a mismatch fails loudly at the first request rather than running slow.
3. Prefix retention at 24 entries. Free in pool terms, and the difference between a 169 s and a 0.8 s response on a repeated long context.
4. Explicit RDMA device selection on multi-port hosts. The first bring-up failed every request with a closed control plane. The expert ranks were creating their queue pairs on the first RDMA device the host enumerated, which carried no IPv4 RoCEv2 address; the fabric address lived on the second port, so the queue-pair transition to ready-to-receive failed. Host-to-host bandwidth tests passed because they bind by device name, which isolated the fault to device selection. The engine has an undocumented map for exactly this, DS41RT_PROTOCOL_V2_VERBS_HOST_DEVICE_MAP (local-ip=device entries), consumed for both client and server endpoints. Setting it on every rank and the coordinator fixed it, and the launch scripts now refuse to start without a fabric address.

Provenance. This page was first published at capacity 256. The widening to 1024 was suggested by tj, DS41RT's author, on the grounds that for most agent workloads the pool was oversized and the prefill lane was what mattered; it was measured the same day (679k-token cold prefill 397 s to 169 s, decode 73.5 to 74.1 tok/s) and became the default.

7 · What is and is not established

• ★ The baseline column is borrowed, and the like-for-like run is pending. Section 2 compares against the reference author's published numbers. Our own run of the reference TP4 stack on the same four Sparks, with the same prompt ladder and bench, has not been done yet; it is the next measurement, and section 2 will be rewritten from it.
• Single runs. Each ladder rung and concurrency probe is one request or one batch. Engine variance on this class of hardware is typically 5 to 10%, and the Sparks are known to have a GPU slow state that can move a single decode cell by up to about 1.5×.
• The host tier was not a bottleneck. Dropping the page cache changed neither prefill (2,100 vs 1,913 tok/s at 26k tokens at capacity 256, within noise) nor decode. A faster local tier for the Engram tables would matter only at higher prefill capacity or under multi-session churn, and neither was reachable on this card with the drafter on.
• Prompt kind matters for decode. 74 tok/s is an essay prompt; 85 is code and reasoning. Quote the one that matches your workload.
• Two results on this page come from the earlier capacity-256 configuration: the concurrency shapes in section 5 and the standard bench in section 4. Decode is independent of the prefill capacity (measured), so the bench stands; the concurrency shapes are pool-bound at the new default and are queued for a re-run.

8 · Reproduce

Everything needed is public. The engine is unmodified upstream; our contribution is how to build it for a 5090 and how to launch and gate the fleet.

# build (on the 5090 host; the AOT export reads this GPU's SM count):
REPO=/path/to/ds41rt scripts/afd-build-coordinator.sh        # -> ds41rt-coordinator-rtx5090:v1
# prove it fits and boots with no Sparks:
MODEL_DIR=/path/to/DeepSeek-V4.1-Flash scripts/afd-ready-probe.sh   # READY in ~4 s
# operating point (scripts/afd-launch-coordinator.sh):
#   prefill capacity 1024 (batch 1024; BATCH=80 selects the pool-first capacity 256),
#   dSpark on, concurrency 16, prefix-cache entries 24, max context 1,048,576,
#   memory reservation 97%
# experts (scripts/afd-launch-experts.sh, per Spark, worker-first 3,2,1,0):
#   published arm64 image, capacity 1024 (must match the coordinator), ~100 GiB device budget
# fabric: on every rank and the coordinator,
#   DS41RT_PROTOCOL_V2_VERBS_HOST_DEVICE_MAP=<fabric-ip>=<rdma-device>  (see docs/GATES.md)
# measure: unique salted prompts, temperature 0, streaming;
#   prefill = prompt tokens / TTFT; decode = output tokens / time after first token

Sydney timestamps throughout.

9 · Credits

• tpurtell (tj) — the DS41RT engine, its container images, the AFD architecture this deployment runs, and the prefill-capacity suggestion in section 6.
• DeepSeek — the DeepSeek-V4.1-Flash checkpoint.
• tonyd2wild — the TP4 DGX-Spark vLLM reference that is the baseline in section 2.
• Local Inference Labs — the standard benchmark.

The 5090 port, the bring-up diagnostics and every measurement on this page are Turquoise Bay AI's work.