DeepSeek Open-Sources DeepInfer: 155T Daily Tokens on Single Cluster

DeepSeek has open-sourced its high-performance cloud inference framework, DeepInfer, along with the accompanying token encoding/decoding component, deepseek-recipe. These tools currently support online services for the V4.1 Flash model, handling an average of 155 trillion tokens per day.

The framework supports cross-vendor hardware (NVIDIA/Ascend), distributed inference, speculative decoding, and other cutting-edge optimizations. It achieves a throughput of 155T tokens/day, generating $38/hour in revenue per GPU. The deepseek-recipe component ensures precise alignment between API protocols and model tokens.

This content is suitable for large language model inference system engineers, AI infrastructure architects, and cloud service SREs.

On October 20, DeepSeek announced the official open-source release of DeepInfer, a large-scale cloud service inference framework. This framework is already being used for large-scale online inference services for DeepSeek V4.1 Flash.

According to the introduction, DeepInfer integrates high-performance operators and deeply optimized inference engines. It supports capabilities such as distributed inference, speculative decoding, SWA Bounded Replay, and structured output, and can run on both NVIDIA and Huawei Ascend platforms. According to data published by DeepSeek, the single-cluster service throughput based on this system has reached 155 trillion tokens per day. Calculated at off-peak pricing, the overall revenue reaches $38 per hour per GPU.

DeepSeek also open-sourced deepseek-recipe, a production-grade token encoding/decoding component designed for efficient and accurate conversion between mainstream LLM API protocols and model tokens. Its processing logic is fully aligned with the DeepSeek official API, enabling third-party deployments to achieve an API experience as consistent as possible with the official service.

Open Source Repositories:

DeepInfer: https://github.com/deepseek-ai/DeepInfer

deepseek-recipe: https://github.com/deepseek-ai/deepseek-recipe

However, this does not represent a complete open-source version of all of DeepSeek's online inference infrastructure. DeepSeek explicitly stated that KVCache management and load balancing components are not yet fully open-sourced due to their deep coupling with internal infrastructure. The focus of this disclosure is on the overall architecture and key implementation details of the V4.1 Flash inference system, as well as the design logic of these components in real-world large-scale online services.

Actual statistical data from the DeepSeek-V4.1-Flash online system

Prefill on Standard 8-GPU Machines, Decode on 64–128 GPU Super Nodes

DeepSeek explained that three core goals were established during the initial design of the V4.1 Flash inference system: compatibility with compute chips from different vendors; organizing Prefill and Decode around common hardware configurations; and ensuring KVCache retention for over 72 hours to reduce redundant computation.

In specific deployments, DeepSeek uses standard 8-GPU servers to handle Prefill and 64–128 GPU super nodes to handle Decode. These two stages are no longer treated as identical computational loads; instead, computational, communication, and scheduling strategies are designed separately according to their distinct characteristics.

This architecture continues some designs from the DeepSeek-V3 and R1 era but further separates the deployment of visual encoders, Prefill, and Decode. This allows resources to be allocated independently across the three stages, optimizing Time To First Token (TTFT) and Time Between Tokens (TBT) separately. For the Mixture-of-Experts (MoE) component, Prefill typically employs 2–8 way expert parallelism and data parallelism, while Decode expands to 64–128 way Wide-EP to improve GPU matrix computation efficiency through larger batch sizes.

One particularly notable change is that inference system optimizations increasingly focus on how the entire cluster can minimize idle time.

After PD separation, the ratio of resource requirements for the two types changes in real-time with online traffic. If fixed ratios are set in advance, queuing occurs during certain periods while machines of the other type remain idle. Therefore, DeepSeek added automatic scaling mechanisms at the cluster level: the Prefill side determines the number of instances needed based on the count of cache-missed tokens, while the Decode side estimates demand based on the current actual number of computational requests. The scaling process includes upper and lower bounds as well as rate limits for adjustments.

To address resource waste caused by long-tail requests in large EP deployments, Decode instances do not exit immediately upon receiving a shutdown notification. Instead, they enter a one-minute grace period. After the grace period ends, if there are still unfinished requests, the system forwards those requests and their associated KVCaches to other instances in a single batch to continue decoding. Redundant quotas are also reserved during deployment. Some machines remain in a Ready state but are unassigned, serving as inventory for burst scaling; meanwhile, the number of online instances always exceeds theoretical demand. When quota occupancy exceeds 0.8, the system directly prohibits scale-down operations.

Regarding fault tolerance, DeepSeek does not attempt to compress instance restart times to extremely short durations. Actual error-triggered restarts still take minutes. Overall fault tolerance relies on a combination of "redundant quotas + request-level rerouting."

In large-scale EP deployments, all GPUs within an instance form a single communication domain. If any card fails, the entire instance may become unable to continue working. Upon restart, to continue utilizing other healthy GPUs, the platform recounts the available cards and re-determines the EP Size based on the actual number of healthy GPUs obtained.

Data parallelism, expert parallelism, Prefill, Decode, and vision encoding each require distinct load balancers. Consequently, DeepSeek developed separate components: the Prefill Load Balancer, Decode Load Balancer, Vision Encoder Load Balancer, and Expert-Parallel Load Balancer.

For instance, Prefill tasks are categorized into four tiers based on input length—under 16K, 16K–64K, 64K–256K, and over 256K—and engine allocation is dynamically adjusted according to actual computational load. Meanwhile, Decode operations monitor both request volume and total KV Cache length, distributing long and short requests across different devices to minimize throughput losses caused by device waiting times.

Expert parallelism also employs dynamic adjustment. In addition to resident experts, each GPU hosts redundant copies of popular experts as backups. These replicas are reallocated based on recent load metrics for different experts. According to DeepSeek’s production data, load disparities among compute cards in Prefill scenarios have been significantly reduced.

Dedicated 100 PiB SSD Storage for KV Cache with Up to 7-Day Retention

Another major focus within DeepInfer is the KV Cache.

As context windows expand and Agents continuously carry historical information and tool invocation records, the cost of repeated Prefilling rises rapidly. To address this, DeepSeek evolved the KV Cache from a standard VRAM cache into a cluster-level shared data layer.

DeepSeek divides KV Cache storage into Block KV Cache and State KV Cache. The Block KV Cache is managed via paging in blocks of 128 tokens, with persistent data written to an independent 3FS cluster. Simultaneously, the system maintains a separate SWA (Sliding Window Attention) KV Cache. For the first 20 layers, each has a ring buffer of 134 tokens, featuring a window length of 128 tokens plus 6 reserved tokens for speculative rollback. Active Block and State KV Caches reside in inference card VRAM, while the SWA KV Cache is stored in the memory of the machine that generated it (Prefill or Decode), bypassing 3FS persistent storage.

Active Block and State KV Caches are kept in inference card VRAM and managed by Prefill/Decode instances. The SWA KV Cache remains in the host memory of the originating Prefill or Decode machine; it is not written to 3FS, retains only a single replica, and is fetched via 400G VLAN if hit across machines. Paged Block KV Cache is stored in the independent 3FS storage cluster, where 3FS handles data redundancy and kvcache-store manages access.

Rather than using common multi-tier caching, DeepSeek places persistent Block KV Cache directly into an independent storage cluster. This allows all Prefill machines to hit any prefix without relying on prefix-affinity routing, avoiding conflicts between cache routing and load balancing. Additionally, storage and compute nodes can scale independently, and failures in compute nodes do not result in the loss of persistent KV Cache. DeepSeek notes that compute node failure rates are orders of magnitude higher than those of storage nodes, so isolating the cache enhances overall stability.

Currently, 3FS provides approximately 100 PiB of SSD capacity for KV Cache. The Time-To-Live (TTL) for Agent traffic KV Cache reaches 3 days, while Chatbot traffic extends to 7 days. The latter exhibits a more pronounced long-tail hit pattern: a small portion of KV Cache entries may be read repeatedly and frequently. Premature deletion of these entries would significantly impact the overall hit rate. Based on DeepSeek’s experience, reducing the Chatbot KV Cache TTL to around 3 days causes the overall hit rate to drop by 0.5%–1%.

Regarding bandwidth, 3FS is specifically optimized for random reads and writes of KV Cache blocks at a granularity of approximately 112 KiB. Peak aggregate read bandwidth reaches 10 TiB/s, while peak aggregate write bandwidth is about 850 GiB/s.

DeepSeek further breaks down the KV Cache hit path.

Block KV Cache is persisted by Decode instances, ensuring that data generated during either Prefill or Decode phases can be reused by subsequent requests. The entire hit process consists of three stages, with only the first stage lying on the critical path of the request:

1. Matching: The Entrypoint performs prefix matching via hash chains in the local kvcache-store acting in the CHECK role. This step queries metadata only. However, because the hit length must be determined before routing, it cannot be overlapped with other computations.
2. Fetching KV: The system asynchronously moves KV Cache from 3FS into host shared memory via an independent process.
3. Swapping In: Threads within the Engine process transfer data from shared memory to HBM. The goal is to overlap layer-by-layer Host-to-Device (H2D) transfers with forward computation. Since CUDA Copy Engine performs poorly with fine-grained KV Cache block replication, DeepSeek switched to GDR (GPU Direct RDMA). This allows RDMA network cards to saturate PCIe bandwidth without occupying SMs (Streaming Multiprocessors).

In terms of KV Cache management, DeepSeek V4.1 Flash introduced a change from the design used in V4.

Previously, the system wrote KV Cache corresponding to Sliding Window Attention (SWA) to persistent storage to enable precise recovery when needed. However, after deployment, DeepSeek observed that such caches are typically reused repeatedly only within minutes during a single active session. This behavior differs fundamentally from Block KV Cache, which can be hit repeatedly over several days. Continuing to store these minute-level data points on large amounts of SSD space would consume nearly half of the persistent cache capacity without yielding proportional hit-rate benefits.

Therefore, V4.1 adopts SWA Bounded Replay: instead of persisting the SWA KV Cache to SSD for long-term storage, it recomputes only the most recent 128 tokens upon a cache miss, constraining the recomputation scope within a fixed window. This ensures that recomputation cost no longer scales continuously with full context length and model depth, becoming an approximately fixed overhead instead.

DeepSeek states that after adopting this method in the Decoder, each Prefill step requires recomputing only the last 128 tokens of the prompt. Compared to the previous approach of precisely reconstructing large amounts of state layer by layer, the total Prefill computation is nearly halved.

Speculative Decoding Acceleration Continues: No Longer Fixed at Verifying 5 Tokens

DeepInfer also integrates DeepSeek’s speculative decoding framework, DSpark.

Traditional speculative decoding typically generates a fixed number of candidate tokens at once, which are then uniformly verified by the main model. However, the acceptance probability of candidate tokens varies across different positions. If verification is fixed at 5 tokens, significant computational resources may be wasted on positions where candidates are unlikely to be accepted.

DSpark introduces confidence prediction, dynamically determining how many tokens need verification based on the estimated acceptance rate of each candidate token. In actual online deployments, the system still generates 5 candidate tokens at once but selects the subset with the highest expected benefit for verification, based on hardware throughput curves and estimated acceptance rates.

According to results provided by DeepSeek, compared to fixed verification of 5 tokens, variable-length verification improves throughput by 6% in real-world online deployments.

Additionally, Decode scheduling employs asynchronous pipelining between GPU and CPU. In traditional workflows, the GPU might remain idle while the CPU performs scheduling for the next round. DeepInfer allows the CPU to complete scheduling for the next round ahead of time while the GPU executes forward computation and sampling for the current round, maximizing overlap between computation and scheduling to reduce GPU idle time.

Open-Sourcing the API-to-Token Layer

Beyond the core inference engine, DeepSeek has also open-sourced deepseek-recipe, a production-grade component that previously received little attention.

It handles conversion between mainstream LLM API protocols and the actual token sequences used by DeepSeek models. Currently supporting Chat Completions API, Responses API, and Messages API, it unifies different protocols into an internal dialogue format and then generates corresponding prompts and token sequences based on the model version. Request transformation, prompt encoding, and output parsing are primarily implemented in Rust, with Python bindings provided.

DeepSeek specifically addresses special token injection issues. For example, even if user input contains <｜User｜>, <｜image｜>, or similar tool-call tags, they are not directly interpreted as control tokens with special semantics. Instead, they are encoded as plain text to prevent special token strings in user input from being assigned control semantics, which could otherwise affect model inference behavior.

In long-context scenarios, DeepSeek leverages known boundaries of special tokens to tokenize different plain-text segments in parallel before recombining them, reducing the impact of extremely long inputs on CPU usage and Time-To-First-Token (TTFT) latency.

For streaming output, DeepInfer maintains a cross-chunk streaming parsing state machine. Even if a UTF-8 character, tool-call parameter, or structural tag is split across multiple data chunks, the system waits for data completeness before parsing, while returning confirmed portions as early as possible to minimize additional latency during tool calls and long-parameter generation.

For more detailed information, please refer to the official documentation:

https://zhuanlan.zhihu.com/p/2095560295930508953