DeepSeek shares the engine behind its AI service, but only part of it

Chinese AI company DeepSeek released DeepInfer on GitHub on October 20, 2026, a kind of engine for running AI models. It is licensed under Apache-2.0, an open license.

Such an engine is called an inference engine. It takes a fully trained AI model and makes it deliver answers to users. It is the part that determines how fast and how cheaply a chatbot responds.

What's included

The public version supports the DeepSeek-V4.1-Flash model. It runs on Nvidia SM100 graphics chips and Huawei Ascend 950 chips.

The release includes optimized code for distributing work across many chips, as well as speculative decoding. That is a method in which the model guesses several words ahead to save time. It also includes enforcement of output formats, so responses follow a specific structure. DeepSeek has also released a tokenization tool, deepseek-recipe. Tokenization means splitting text into the small pieces the model works with.

According to OrcaRouter, which wrote a technical review, DeepInfer also includes the FlashMLA, DeepGEMM and DeepEP kernel libraries, as well as asynchronous batch scheduling and support for structured output such as JSON schema.

What's missing

The Next Web reports that the release has no storage system for the KV cache, which is used to remember earlier parts of a conversation. It also lacks separation of prefill and decode, two stages in the response process, and has no load balancing across data-parallel replicas.

OrcaRouter assesses that DeepInfer is a reference implementation, in other words an example of the compute path. It was released so others can study and reuse the work on kernels and scheduling. According to OrcaRouter, it is not something you install and point traffic at. According to the same source, the engine requires at least 64 decode GPUs to deliver efficiency gains, and the repository had no versioned releases at the time of publication.

Built on others' work

DeepInfer was originally built primarily on vLLM, with inspiration and contributions drawn from SGLang and Nvidia's TensorRT-LLM. The vLLM project acknowledged DeepSeek on X and promised to incorporate the improvements.

DeepSeek itself says the goal is to run DeepSeek-V4.1-Flash at large scale with high throughput and low latency, and to share the production stack so the community can reduce compute costs and energy consumption. According to The Next Web, DeepSeek's speed figures have not been independently verified.

SGLang also has a guide for running the model. According to it, DeepSeek-V4.1 Flash is a model with 40 layers and 384 routed experts, of which 6 are selected at a time, plus one shared expert. According to a summary of the page, support in an official SGLang release did not yet exist, but there was a preview Docker image.