155 Trillion Tokens a Day, $38 per GPU-Hour: DeepSeek Open-Sources DeepInfer and Lays Bare the Margin Math of LLM Inference
155 Trillion Tokens a Day, $38 per GPU-Hour: DeepSeek Open-Sources DeepInfer
Part One: From “Open Models” to “Open the Whole Pipeline”
On the afternoon of October 20, 2026, DeepSeek did something that would have seemed unthinkable not long ago: it open-sourced DeepInfer, the inference framework actually running its large-scale production environment, under the Apache 2.0 license, together with a production-grade token codec component called deepseek-recipe. The repositories live at github.com/deepseek-ai/DeepInfer and github.com/deepseek-ai/deepseek-recipe.
For the past two years, “open source” in the LLM world mostly stopped at the weights. You could download a trained brain, but how a company connected that brain to tens of millions of concurrent users efficiently was a closely guarded secret. DeepInfer opens the entire engineering pipeline of “how the brain takes orders efficiently.”
More striking, DeepSeek published a set of cold production numbers that, for the first time, put a hard dollar figure on efficient inference for DeepSeek V4.1 Flash:
- A single cluster serves 155 trillion tokens per day;
- The average request carries 77K input tokens, yet time-to-first-token (TTFT) averages just 400 ms;
- Decoding runs at 230 tokens/second, up to 300 tokens/second in coding workloads;
- 96% of inputs hit the cache on average and never need recomputation;
- At off-peak (“valley”) API pricing, revenue is $38 per GPU-hour (roughly RMB 254).
That last figure was called “a nuclear bomb” on Hacker News. Some estimated an 82% gross margin: rent a B300 GPU for about $7/hour, and as long as utilization is high, a healthy spread remains after rent. In blunt terms it told the market that LLM inference is not a loss-leader: with idle-free hardware and smart scheduling, low API prices and solid margins can coexist.
This article is about the engineering, not the business story. How does DeepInfer minimize GPU idle time, cut redundant computation, and optimize latency in pieces? Which techniques are universal lessons worth copying, and which are so deeply welded to DeepSeek’s internal infrastructure that they are not in the open-source release at all?
Here is the overall architecture; we then go through it block by block.
User requests (Chat / Responses / Messages API)
|
v
+--------------------------------------+
| deepseek-recipe codec layer |
| special-token boundary detection / |
| parallel chunked text encoding / |
| streaming UTF-8 state machine / |
| direct JSON-fragment emission |
+--------------------------------------+
|
+-----------------+------------------+
| |
v v
+------------------+ +---------------------+
| Prefill cluster | | Decode super-node |
| 8-GPU nodes | | 64-128 GPUs |
| EP2~8 / DP2~8 | | EP64~128 Wide-EP |
| optimize TTFT | | optimize TBT |
+------------------+ +---------------------+
| |
| expert parallelism / dynamic LB / |
| autoscaling instance ratio |
+-----------------+-------------------+
|
v
+--------------------------------------+
| shared KVCache store (3FS ~100 PiB) |
| 128-token blocks / 4 mainKV layers |
| TTL 3-7 days / read 10TiB/s |
| write 850GiB/s / any-prefix hit |
+--------------------------------------+
|
+-----------------+------------------+
| DSpark speculative decoding |
| SWA Bounded Replay |
| confidence-based variable verify |
| (+4-7% throughput) |
+--------------------------------------+
|
NVIDIA SM100 GPU / Huawei Ascend 950 NPU
Part Two: Three Optimization Lines — Throughput, Cost, Latency
To understand DeepInfer, first accept that LLM inference at scale is three competing goals. DeepSeek’s technical report on the V4.1-Flash-1017 inference system organizes them into three independent “lines”:
- Throughput line: keep GPUs from idling and emit as many useful tokens as possible;
- Cost line: minimize redundant prefill — reuse wherever possible;
- Latency line: TTFT and inter-token interval (TBT) are different workloads and must be optimized separately.
A user sends a request and waits for an answer. The server, meanwhile, shares a pool of GPUs across thousands of requests, reads enormous weights from HBM, schedules cross-device compute and communication, manages the KV cache, and controls queueing and generation latency. With the same model, different inference implementations can differ by several-fold, even an order of magnitude, in how many users each card serves and how long they wait. The inference framework exists to close that gap.
DeepInfer is Python-based, initially built largely on vLLM, while borrowing ideas from SGLang and TensorRT-LLM. It is not a rewrite from scratch; it stitches best-of-breed practice together with DeepSeek’s own kernels: FlashMLA (attention), DeepGEMM (GEMM), DeepEP (expert-parallel all-to-all communication), TileKernels, DeepSelect, and the self-developed TileLang compilation kernel, covering the hottest paths of attention, matrix multiplication, and communication.
We begin with the hardware-organization philosophy: the Prefill/Decode split.
2.1 Why Prefill and Decode Must Be Separated
A request has two fundamentally different phases:
- Prefill reads the entire prompt (77K tokens on average) at once. It is compute-bound, like swallowing a book in one batch.
- Decode then emits one token at a time; each step reads all weights but produces one token. It is memory-bound, like spelling out a sentence word by word.
Mixing them on the same cards hurts both: Prefill’s big compute blocks Decode’s rhythm (users feel stutter), while Decode’s low arithmetic intensity wastes the compute Prefill could saturate.
DeepSeek deploys them on physically separate hardware, with opposite parallelism settings:
Phase Hardware Parallelism Target
-----------------------------------------------------------------
Prefill 8-GPU nodes EP2~8 / DP2~8 (small) TTFT
(small if HBM fits) finish on one node big batch, GEMM efficiency
Decode 64-128 GPU nodes EP64~128 / DP64~128 TBT
(go wide) Wide-EP fewer params/card, more requests
The counterintuitive “small for Prefill, large for Decode” rests on V4.1 Flash being a MoE model with 384 routed experts. In Decode, DeepSeek spreads the 384 experts across 64 cards — 6 experts per card, plus 1 replica of a hot expert, for 7 total. The fewer experts resident on a card, the less weight memory traffic per generated token, and the more concurrent requests the same fleet can absorb. In offline tests, a 64-card Decode group produced over 26,000 tokens/second per card at peak.
Part Three: Expert Parallelism and Four Load Balancers
The hardest MoE inference problem is imbalance: in real traffic, “hot experts” get slammed while cold ones sit idle. Without balancing, a few hotspot cards become the bottleneck of the whole super-node.
DeepInfer uses four load balancers, one per workload type; the four-level Prefill schedulers share a single “cost model,” and every decision ultimately lands on a unified compute-quota:
Workload Balancing strategy moe_lb
-----------------------------------------------------------------------
Prefill four-level scheduling, shared cost model EP2/DP2: 1.0320
EP4/DP4: 1.0618
Decode two-dimensional (expert + pipeline) 1.090
Vision Encoder patch-granularity allocation (single card) --
EP redundant expert params + dynamic reflow --
moe_lb measures balance, 1.0 being perfectly even. Prefill at small parallelism reaches 1.03; Decode, with a larger, more random super-node, relaxes to 1.09 — both within a tight band. EP balancing uses “redundant expert parameters plus dynamic reflow”: replicas of hot experts are pre-provisioned (that seventh expert per card), and expert placement is adjusted at runtime by live load.
Here is simplified Python illustrating why replicating hot experts tames tail latency:
import numpy as np
class ExpertPlacement:
"""Toy model of expert placement and balancing in MoE Decode."""
def __init__(self, n_experts=384, n_cards=64, replica_budget=64):
self.n_experts = n_experts
self.n_cards = n_cards
# 384 experts spread evenly over 64 cards, 6 per card
self.placement = {e: e % n_cards for e in range(n_experts)}
self.replica_budget = replica_budget # extra replicas (1 per card)
self.replicas = {} # expert -> replica card
def add_hot_replicas(self, hotness):
"""Replicate the hottest experts to spread hotspot traffic."""
ranked = np.argsort(hotness)[::-1]
for i, expert in enumerate(ranked[:self.replica_budget]):
self.replicas[int(expert)] = i % self.n_cards
def card_load(self, token_experts, hotness):
"""Load per card after routing, with replica spillover."""
load = np.zeros(self.n_cards)
for e in token_experts:
primary = self.placement[e]
if e in self.replicas:
rep = self.replicas[e]
card = rep if load[rep] < load[primary] else primary # least-loaded
else:
card = primary
load[card] += hotness[e]
return load
# Zipf-skewed hotness: a few experts take most tokens
rng = np.random.default_rng(0)
hotness = rng.zipf(1.3, size=384).astype(float)
tokens = rng.choice(384, size=100000, p=hotness / hotness.sum())
pl = ExpertPlacement()
before = pl.card_load(tokens, hotness)
pl.add_hot_replicas(hotness)
after = pl.card_load(tokens, hotness)
def imbalance(load):
return load.max() / load.mean()
print(f"no replicas, max/mean imbalance: {imbalance(before):.3f}")
print(f"with hot replicas: {imbalance(after):.3f}")
print(f"hotspot peak load reduction: {(1-after.max()/before.max())*100:.1f}%")
Running this shows that giving the 64 hottest experts one replica each and spilling requests to the less-loaded copy materially cuts hotspot peak load — the micro-principle behind DeepSeek’s “redundant experts + dynamic reflow” protecting Decode tail latency. This is a teaching toy; the real system also weighs cross-card communication (the DeepEP all-to-all) and super-node topology.
Part Four: The Cost Line — Crushing Redundant Prefill
If throughput is about organizing hardware, cost is about never recomputing twice. This is the densest engineering and the best explanation for the 96% cache-hit rate.
4.1 Long-Lived KVCache: 3 or 7 Days, Not 5 Minutes
Long conversations and agents have a clear signature: after a tool call, the next request often carries the same system prompt, tool definitions, and history again. Recomputing those prefixes every time is a huge waste.
Most major providers (Anthropic, OpenAI, Google) keep KVCache for 5 minutes to 24 hours. DeepSeek targets 72+ hours, with online TTLs of:
- 3 days for agent traffic;
- 7 days for chatbot traffic.
An ablation cut the chatbot TTL from 7 days to 3 and saw hit rate drop 0.5%-1%. At hundreds-of-trillions-of-tokens scale, that fraction is a fortune in compute.
4.2 Block KVCache Off the Node: a 100-PiB 3FS Cluster
The obstacle to long retention is size: the KVCache is far too big for node HBM/DRAM, and if a cache lives on a specific Prefill node, only requests routed there hit it (“prefix-affinity routing”), which shackles scheduling.
DeepSeek’s aggressive answer: block KVCache never enters node memory. It is cut into 128-token blocks, organized into 4 mainKV layers, and placed on a dedicated 3FS distributed storage cluster:
+-------------------------------------------------------------+
| 3FS shared KVCache cluster (~100 PiB SSD) |
| |
| read bandwidth ~10 TiB/s write ~850 GiB/s |
| random I/O at ~112 KiB block granularity |
| |
| any Prefill node -> can hit ANY prefix |
| (cost: drop prefix-affinity routing; gain: global hits) |
+-------------------------------------------------------------+
^ ^ ^ ^
Prefill node A Prefill node B Prefill node C Prefill node D
Putting cache on a shared SSD cluster with 10 TiB/s read bandwidth yields “any Prefill machine can hit any prefix,” so the scheduler never has to pin a request to a fixed machine for a cache hit. Distributed random-read capability is traded for scheduling freedom and maximal hit rate.
4.3 SWA and Bounded Replay: Recompute Only the Last 128 Tokens
V4.1 also has sliding-window attention (SWA), a 20-layer x 134 ring. Its nature implies a different policy: it stays only in node memory, is not persisted, has a TTL of about 180 seconds, and still achieves 98%+ hits. On a miss, Bounded Replay recomputes.
A key lesson from V4: it once tried to persist SWA to disk, which consumed nearly half of the persistent VCache capacity without gaining commensurate hits — sliding-window attention only looks at recent context, so long-term caching adds little. V4.1 switched to Bounded Replay: on a miss, recompute only the last 128 tokens, nearly halving Decoder-side prefill computation.
It is a classic engineering call: not all KV deserves persistence; for “look-at-recent” attention, recomputing a short slice beats storing a long span.
Part Five: The Latency Line — Split TTFT from TBT, Then Speculate
5.1 Three-Way Decoupling: Encoder / Prefill / Decode
The hardware split already sets up latency work: Prefill resources optimize TTFT, Decode resources optimize TBT. Several easy-to-miss, body-feel details:
- Chunk size solved backward from a time budget: micro-batch size is derived from the target latency, not guessed;
- Tokenization must not stall TTFT: it goes from linear to parallel chunked encoding — plain text between special tokens is encoded independently and concatenated; the hot path is in Rust;
- Stream while generating: a cross-chunk state machine buffers incomplete UTF-8 code points, maintains per-stage (reasoning/body/tool-args) state, and uses KMP to incrementally match delimiters without backtracking over finalized text;
- Tool arguments are not built whole: JSON fragments are emitted directly along DSML boundaries, with strings escaped piece by piece, shortening the wait for the first argument fragment.
These byte-level frugality efforts jointly hold average TTFT to 400 ms.
5.2 DSpark Speculative Decoding: Confidence Sets the Verify Length
Speculative decoding uses a small “draft model” to guess a run of candidate tokens, then has the main model verify them in one parallel pass; accepted tokens are adopted for free. The classic approach fixes verification at 5 candidates.
DeepSeek’s DSpark goes further: a Confidence Head estimates each candidate’s acceptance probability and dynamically sets the verify length — verify more when the draft is confident, fewer when not, avoiding verification spend on candidates bound to be rejected. A low-rank logit bias corrects systematic draft/main-model deviation.
Versus fixed-5 verification, variable-length verification yields about 4%-7% (one official figure: 6%) throughput at the production operating point. A toy illustration:
def fixed_verify_accept(accept_probs, k=5):
"""Fixed k: always spend k verification units regardless of confidence."""
accepted, cost = 0, 0
i = 0
while i < len(accept_probs):
window = accept_probs[i:i+k]
cost += k
run = 0
for p in window:
if p >= 0.5:
run += 1
else:
break
accepted += run
i += max(run, 1)
return accepted, cost
def adaptive_verify_accept(accept_probs):
"""Variable length: verify more when confident, less when not."""
accepted, cost = 0, 0
i = 0
while i < len(accept_probs):
window = accept_probs[i:]
v, cum = 0, 1.0
for p in window:
v += 1
cum *= p
if cum < 0.25 or v >= 8: # stop if joint confidence is low / cap
break
v = max(v, 1)
cost += v
run = 0
for p in accept_probs[i:i+v]:
if p >= 0.5:
run += 1
else:
break
accepted += run
i += max(run, 1)
return accepted, cost
rng = np.random.default_rng(42)
probs = rng.beta(2, 2, size=2000)
a1, c1 = fixed_verify_accept(probs.tolist(), k=5)
a2, c2 = adaptive_verify_accept(probs.tolist())
print(f"fixed: accepted {a1:5d}, verify-cost {c1:6d}, yield {a1/c1:.3f}")
print(f"adaptive: accepted {a2:5d}, verify-cost {c2:6d}, yield {a2/c2:.3f}")
print(f"per-unit verify throughput gain: {(a2/c2/(a1/c1)-1)*100:.1f}%")
The intuition: verifying a candidate that will be rejected is pure waste. Letting verify length follow draft confidence spends the same verification budget where it pays. DeepInfer also overlaps CPU-side scheduling with GPU execution to fill the gaps between Decode steps (pipeline overlap), and relies for fault tolerance not on fast restarts but on spare machines, request rerouting, and degraded restarts.
Part Six: Dual Hardware, deepseek-recipe, and What Is Not Open
6.1 NVIDIA + Huawei Ascend
DeepInfer explicitly supports two stacks:
- NVIDIA: CUDA 13 and SM100-architecture (Blackwell-class) GPUs;
- Huawei Ascend: CANN 9.2 and Ascend 950 NPUs.
You select the accelerator environment variable at build time. This continues DeepSeek’s multi-vendor silicon line — a company deeply tied to domestic compute from training through inference now also publishes the methodology for running extreme inference on domestic NPUs.
6.2 deepseek-recipe: Behavioral Consistency for Third-Party Deploys
The same model often “feels different” between the official API and third-party deployments. Many differences hide in the unsexy but critical details of request codec, streaming parsing, and tool-call formatting. deepseek-recipe is the production codec that fixes this:
- Supports Chat Completions, Responses, and Messages APIs;
- Uses special-token boundary detection, parallel chunked text encoding, and stream parsing for long-text efficiency;
- Keeps third-party API handling aligned with the official service, covering reasoning content, tool calls, and streaming.
For enterprises plugging DeepSeek into existing agent workflows, this layer is worth as much as the engine — it determines whether switching servers causes behavioral drift.
6.3 Honesty: What the Open-Source Release Lacks
DeepSeek did not dump the whole online system. It explicitly states that KVCache management and load balancing are not fully open-sourced because they are deeply coupled to internal infrastructure (the 3FS cluster, super-node networking, the unified quota system). The repository provides:
- The core engine for offline inference and large-scale serving (optimized kernels, parallelism, speculative decoding, structured output, async scheduling);
- But to build a full production service on par with the official one, you still must supply the KVCache system, vision encoding, Prefill/Decode separated deployment, and cross-replica load balancing.
DeepSeek open-sourced the engine blueprints, but the drivetrain and the road-map database you must build yourself. It is a pragmatic boundary — honoring openness without forcibly extracting pieces welded to its own data center that no one else could run anyway.
Part Seven: The Economics, and What It Means for the Industry
Back to the headline number. At $38 per GPU-hour and roughly $7/hour to rent a B300, the market inferred gross margins near 80%. A cross-check: The Information last month put DeepSeek’s annualized revenue at about $1 billion (~$2.74M/day); at $38/GPU-hour that maps to only about 3,000 GPUs.
Both figures deserve caution: $38 is an idealized off-peak, high-utilization figure, and the 82% margin deducts only GPU rent — not lifecycle training spend, data, R&D headcount, or data-center build-out. Hacker News debated exactly this — “AI has no bubble, inference is a great business” versus “count the whole lifecycle cost.”
But whatever the precise accounting, DeepInfer’s open release sends three firm signals:
- Inference efficiency is a core competency, not an afterthought to model quality. As architectures converge, this systems engineering decides how many users a unit of compute serves and how low you can price while still profiting.
- Open source is pushing from weights to the full stack. Models, kernel libraries (TileLang), the inference engine (DeepInfer), and the codec (recipe) open in layers, systematically lowering the barrier for private/public AI services.
- Extreme efficiency supports both low prices and profit, reshaping the API price curve. Once everyone can see how DeepSeek makes a card take orders cheaply and fast, inference competition only gets more intense — and every developer calling an API wins.
Researcher Deli Chen framed the moment as a contest between two futures, open and closed. For engineers, the concrete good news is simpler: you no longer just use a cheap model — you can see how the machine that swallows 155 trillion tokens a day is built, and bring its Prefill/Decode separation, tiered KVCache reuse, and confidence-based speculative decoding into your own systems.
That may be the most consequential part of DeepInfer going open.
Sources: DeepSeek’s Zhihu technical report on the DeepSeek-V4.1-Flash-1017 inference system; GitHub
deepseek-ai/DeepInferanddeepseek-ai/deepseek-recipe; 36Kr, Zhidx, Jiqizhixin, MarkTechPost, and related reporting (2026-10-20). Code snippets are illustrative, not official implementations.