DeepSeek Open-Sources DeepInfer: $900 Per GPU Per Day

One GPU, one hour—DeepSeek makes $38. That’s over $900 a day per card. This isn’t marketing copy; it’s a direct quote from the official deployment documentation that accompanied DeepSeek’s release of the DeepInfer inference framework on October 20, 2026.

What’s really worth unpacking here isn’t how tempting “$900 a day” sounds, but the previously unseen middle layer behind it: once a model is trained, all you have is a pile of weights. How do you queue requests, batch them, distribute work across hundreds of GPUs, and stream results back—all while keeping per-token economics in the black? That whole layer, along with the deployment Q&A, just got handed over in full.

1. Where the $38 Comes From

The original text from the official deployment Q&A reads:

A single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing).

Translation: a single cluster currently serves 155 trillion tokens per day, generating revenue equivalent to $38 per GPU-hour (at off-peak pricing).

155 trillion—a sense of scale: divide it by 86,400 seconds and the system pushes through roughly 1.8 billion tokens per second. And that’s just one cluster, not DeepSeek’s full production capacity.

But two caveats are in order:

First, this is revenue, not profit. DeepSeek hasn’t disclosed costs, nor how many clusters of this size they’re running. “$900 per GPU per day” is real; how much they actually net is anyone’s guess.

Second, $38 is at off-peak pricing. That makes the figure conservative—the same throughput during peak hours would generate more revenue. We’ll dig into the pricing model in the next section.

2. What DeepInfer Actually Runs

Let’s get the concept straight first. After a model is trained, all you have is weights. To serve tens of millions of users simultaneously, you need an entire layer of systems in between: the infrastructure to queue requests, batch them together, fan them out across hundreds of GPUs, and stream results back.

More concretely: when you send a long prompt, the model stores what it’s already read (this is the KV cache), so the next turn doesn’t have to reprocess everything. How well this layer performs directly determines per-token cost.

In DeepSeek’s own framing, this is a middle layer the public has never really seen: invisible to end users but critical to the operation.

The official architecture diagram looks like this:

Top to bottom:

• The top layer is DeepSeek Recipe—incoming requests are first tokenized here
• Three branches underneath: visual encoding (VE) for images, Prefill to process the input, Decode to generate the response
• Each branch has a load balancer in front to distribute the work
• At the bottom is a storage cluster dedicated to caching (the 3FS distributed file system)

So the DeepInfer repository contains the engine itself: high-performance operators, model and sampling runtime, parallelism strategies (Attention DP / balanced CP, MoE EP + EPLB), speculative decoding, async scheduling, structured output. The repo started from vLLM, with ideas borrowed from SGLang and TensorRT-LLM; it’s Apache-2.0 licensed and currently targets the DeepSeek-V4.1-Flash model.

3. DeepSeek Reverses Itself

The subtext here is what makes this story interesting: a year and a half ago, DeepSeek itself said it wouldn’t open-source the engine.

In April 2025, DeepSeek published “The Path to Open-Sourcing the DeepSeek Inference Engine,” laying out three reasons for keeping the engine proprietary:

Official Reason	Key Points
Codebase Divergence	The code forked from an early vLLM version over a year ago and has diverged too far
Infrastructure Dependencies	Tightly coupled to internal infrastructure, including cluster management tools
Limited Maintenance Bandwidth	We’re a small research team with no bandwidth to maintain a large open-source project

Note a flip-flop in the timeline: in April 2025, DeepSeek previewed open-sourcing the inference engine on GitHub, but by May it reversed course—choosing to upstream core optimizations into vLLM, SGLang, and other open projects rather than releasing the full internal codebase.

A year and a half later, on October 20, 2026, the engine was released after all—and this time, the production-grade tokenization and request-handling component deepseek-recipe came along with it. The latter’s logic is fully aligned with the official DeepSeek API, meaning any third-party deployment will exhibit the same interface behavior as the official one.

The result: more and more vendors adopt DeepSeek’s standard stack, and the ecosystem snowball keeps growing. Within a day of publication, the repo passed 12,000 stars; by the time of writing it’s closing in on 15,000, with 1,400+ forks and the companion deepseek-recipe repo at 1,700+ stars.

4. The Ledger: Off-Peak Pricing + Cache Hits

The “$900 per GPU per day” headline rests on the API pricing structure. DeepSeek’s API runs on a peak/off-peak schedule, with off-peak rates slashed to half price:

• Peak hours: weekdays 01:00–04:00 and 06:00–10:00 UTC, i.e., Beijing time 09:00–12:00 and 14:00–18:00 (excluding Chinese statutory holidays)
• All other times (including weekends and holidays) run at off-peak rates.

Official pricing for DeepSeek-V4.1-Flash (model name deepseek-flash):

Per 1M tokens	Off-peak	Peak
Input (cache hit)	$0.003 (~¥0.02)	$0.006 (~¥0.04)
Input (cache miss)	$0.15 (~¥1)	$0.30 (~¥2)
Output	$0.60 (~¥4)	$1.20 (~¥8)

Two observations:

One, the $38 figure uses off-peak pricing, i.e., the “$0.02 cache-hit input, $4 output per million tokens” tier—it’s the conservative assumption.

Two, that $0.02 only holds up if cache hits dominate. A cache miss on input runs ¥1 per million tokens—50× the hit price. The entire economic case rests on “the same context being reused many times over”—which is also why we’ll spend so much real estate on the cache cluster below.

Also worth noting: on the same price sheet, DeepSeek-V4-Pro’s off-peak rates are $0.022 cache hit, $0.66 cache miss, $1.98 output, with a concurrency cap of 500. Flash’s concurrency cap is 2,500.

5. Ascend 950 Sits Alongside Blackwell

The easiest-to-miss but most strategically loaded detail in this release is the supported hardware list:

Just two entries:

• NVIDIA SM100 (i.e., the Blackwell / B200 generation)
• Huawei Ascend 950 (Ascend 950 NPU)

Pay attention to the signal here: the most widely deployed H100 and H800 are absent from the list. This isn’t a compatibility issue—the system is designed around the latest scale-up domains (super-nodes); the Wide-EP decode architecture starting at EP64 fundamentally requires newer hardware.

For model vendors, this means one fewer single-vendor chokepoint. For domestic compute, Ascend has landed on the same production-grade inference framework support list as Nvidia’s flagship for the first time. That Huawei’s official launch events kept crediting Liang Wenfeng’s team wasn’t just politeness.

6. Don’t Pop the Champagne Yet: Three Things That Weren’t Open-Sourced

“Fully open-source” deserves an asterisk. The README explicitly lists three things needed to build a complete inference service that were not included in this release:

1. KV cache system—the infrastructure that stores what the model has already read
2. The orchestration layer that splits VE / Prefill / Decode across hosts (most of the engine functionality is implemented, but the orchestration layer isn’t shipped)
3. Load balancing between data-parallel replicas

Cross-reference with the architecture diagram from Section 2 and it clicks: the load balancers and the storage cluster at the bottom are largely absent from this codebase. The official rationale: these components are too tightly coupled to internal infrastructure, so only the engine was released—scheduling and caching stay in-house.

The cache layer is substantial, and the official doc’s specifics are worth pulling out:

Item	Value
Storage medium	3FS distributed file system, 10+2 erasure coding, tolerating 2 simultaneous node failures per group
Total KV cache capacity	~100 PiB of SSD
Cache retention	7 days for chat traffic, 3 days for Agent traffic (chat has a longer-tail hit rate; capping it at 3 days would cost 0.5%–1% in hit rate)
Storage-to-compute ratio	~14.4 TB logical per GPU (physical is higher)
Single storage server	20 × 30.73 TB SSDs, ≥40 GB/s read, 5 GB/s write, 24-hour data retention
Prefill context tiers	Four buckets: <16K / 16–64K / 64–256K / ≥256K, engines dynamically allocated by compute budget

Why is cache retention measured in days? Because the same long context tends to be re-queried frequently—the longer you keep it, the less you recompute. That’s how the $0.02 cache-hit price point holds up.

7. The Bar: 128 GPUs Minimum

This is where the cold water really hits. The official Q&A on “minimum deployment unit” is blunt:

1P + 1D is deployable… Since the recommended decode configuration is EP64 or above, the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode.

Translation: 1 Prefill + 1 Decode can deploy, but since the recommended decode configuration is EP64 or above, the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode.

128 Blackwell-class GPUs is a small data center on its own. So:

• If you’re an individual developer with one or two 4090s, this isn’t really for you.
• If you want to run DeepSeek locally, vLLM, SGLang, or one of the quantized builds is far more realistic. For guidance on picking one, see my earlier write-ups: Strata Hands-On: Running a 125B Trillion-Param LLM Locally on 64GB RAM + 16GB GPU and Stop Guessing Which Local LLM to Download: One Command Tells You What Fits Your Rig.

While we’re at it, here’s the parallelism strategy so you can gauge scale:

Stage	Recommended Parallelism
Prefill	EP2–8 / DP2–8, using the smallest EP/DP that HBM allows
Decode	EP64–128 / DP64–128 (Wide-EP), using the largest EP/DP the hardware allows
Visual Encoding (VE)	One instance per card is fine; cheaper cards are recommended

8. Who’s Actually Going to Use This

The answer is straightforward: cloud providers, plus large enterprises that need on-prem deployments.

Third-party DeepSeek deployments used to lean on generic frameworks like vLLM and SGLang, with DeepSeek’s own optimizations trickling out as scattered patches. Now the entire engine is in their hands, and DeepSeek Recipe locks in interface behavior consistent with the official version—so cloud-deployed DeepSeek will get much closer to official-grade efficiency.

The obvious corollary: third-party DeepSeek API pricing likely has room to keep dropping. Lower deployment barriers mean more supply, and price wars become the natural outcome.

9. The Real Treasure Is the Deployment Q&A

If I could only leave you with one takeaway: the code itself isn’t as scarce as you might think—the accompanying deployment Q&A is the truly rare asset.

Because it commits to paper the playbook that cloud providers usually keep under wraps:

• How to split parallelism (the Prefill EP2–8 vs. Decode EP64–128 tradeoff)
• How to balance experts (EPLB: Prefill uses 4 redundant experts per card, 100-step stats window, rebalance every 1,000 steps; Decode uses 1 redundant expert per card, EP64 means 7 experts per card, 1,000-step window, rebalance every 10,000 steps)
• Where the cache lives, how long to keep it, how to evict by LRU
• What to do when a card dies (in large-EP deployments an instance is an entire communication domain; a single card failure takes the whole instance down, and on restart you re-derive the EP size from the actually-healthy card count)
• How good load balancing actually gets today—roughly 80% of the offline oracle baseline of perfect balance

The most operationally useful piece is speculative decoding (DSpark). A smaller model drafts 5 tokens at a time; the large model verifies them. DeepSeek attaches a confidence score to each draft and only verifies the ones that look promising.

The official numbers: draft tokens verified per step drop from 5.00 to 3.2–4.4—roughly 30% fewer; yet the average accepted length stays essentially flat (3.29 vs. 3.20); at the production operating point (~60 concurrent sequences per card), overall throughput climbs 4%–7%.

In isolation those gains look unremarkable. Multiplied across 155 trillion tokens a day, however, what you’re saving is real money on power and GPUs.

10. Three Takeaways

First, DeepSeek just handed part of its pricing power to third parties. The trifecta of engine + Recipe + deployment Q&A converts “near-official efficiency” from tribal knowledge into reproducible engineering. Third-party API prices have further to fall.

Second, Ascend 950 landing on the list carries more strategic weight than the code itself. Domestic compute has, for the first time, secured parity with Nvidia’s flagship on a production-grade inference framework’s support matrix. The conspicuous absence of H100/H800 signals the system was designed from the ground up around “new hardware + new super-nodes,” not backward compatibility for existing fleets.

Third, this is a gift to professional readers, not to individual developers. With a 128-GPU bar, anyone with a 4090 or two should not get distracted by the “$900 a day” headline. But if you care about how to push inference costs down—how to design caches, how to tier prefill, how to slice load balancing, how speculative decoding saves compute—this doc is worth reading end to end. Pair it with 4GB VRAM, 70B Parameters: Dissecting AirLLM’s “Layer Streaming” Magic, which tackles the same cost-optimization problem from the hobbyist angle.

Summary

DeepSeek ran a complete arc over a year and a half—“we won’t open-source it” → “we’ll contribute back differently” → “let’s just open-source the whole thing”—and what it ultimately delivered is not a model but a reproducible production-grade inference pipeline.

Just commit these three numbers to memory: 155 trillion tokens per day, $38 per GPU-hour, minimum efficient deployment 128 GPUs. The first signals scale; the second proves the system makes money; the third tells you who can actually play.

The thing really worth keeping is the methodology behind it: nibble per-token costs down across cache, tiering, load balancing, and speculative decoding. That 4%–7% throughput gain, scaled by 155 trillion tokens per day, is the difference on your power and hardware bills.

Repository: github.com/deepseek-ai/DeepInfer (Apache-2.0)
Original Deployment Q&A: doc/inference_deployment_qa.md
Official API Pricing: api-docs.deepseek.com/quick_start/pricing

Are you using the DeepSeek API or running it locally? How much further do you think third-party API pricing can drop? Drop your own cost numbers in the comments, bookmark this piece, and come back to compare notes when DeepSeek’s next model lands.