On the morning of 20 October 2026 DeepSeek posted four messages and a link. The link is a program. The messages call that program the stack that answers DeepSeek-V4.1-Flash. The download is real. The building that makes the speed claims true is not in it.
Short answer: DeepSeek DeepInfer (the program DeepSeek uses to turn its finished model into the next piece of text) is now public, under Apache 2.0 (a license that lets you use, change, and sell software if you keep the notices). It runs one model, on two kinds of data-center chip, as a script that stops when the answer is done. The memory warehouse, the split between reading machines and writing machines, and the traffic cop are still private. A gaming PC is the wrong machine. So is the export-compliant workstation card in today’s other Nvidia headline.
The same day’s louder money stories were a private funding race and a chip order. This post is the machine. If you only want the headline, the next nine words are enough: they open-sourced the engine, not the service.
What has to be true before the new words mean anything?
A chatbot answer is a guess, made one small piece at a time, by a file of numbers, driven by an ordinary program.
Start below the jargon.
A program (a list of steps a person wrote) does step one, then step two, then stops. It does not invent step forty.
A language model (a system trained to guess the next piece of text) is not that list. It looks at the text so far and offers a continuation. The continuation is not retrieved from a page. It is chosen from what the training made likely.
The thing that was trained is a file of weights (the numbers that store what the training decided). Training (the expensive job that produces that file) is already finished before anyone chats. Inference (using the finished file to guess the next piece) is the job that happens when you send a message. DeepInfer is an inference program. It does not train anything.
A token (the small piece being guessed, often part of a word, sometimes a whole word, sometimes punctuation) is the unit of that guess. You do not need the dictionary of tokens to follow the rest. You need the idea that the model does not write your paragraph in one bite. It writes a token, then another, and the previous ones become part of what it looks at next.
DeepSeek-V4.1-Flash is the file. The model card on Hugging Face lists it as DeepSeek-V4.1-Flash-1017, with 763 billion parameters (the count of those numbers) and a context window (the amount of text it can look at in one go) of 1 million tokens. The card’s license line says MIT. DeepInfer is the program that runs that file. Gluing the two names into one product is how today’s posts get the download wrong.
What landed on 20 October, and who said it?
DeepSeek published the engine and a separate translator, and said future models will land in the same place. It did not publish the rest of the service.
The primary announcement is DeepSeek’s own thread, posted at 09:30 UTC on 20 October 2026. The first message says DeepInfer is “our in-house inference framework, now fully open source,” that it is “powering DeepSeek-V4.1-Flash at scale,” and that the point is to “lower compute costs, reduce energy use, and make AI more affordable.” The second message lists kernels (the small, heavily tuned math routines that actually hit the chip), distributed inference, speculative decoding, SWA bounded replay, and structured outputs. Those get definitions below, in that order, not as a brochure. The third message open-sources deepseek-recipe, “our production token encoding and decoding component,” aligned with the official API. The fourth says future models’ inference code will land in DeepInfer too.
The README on GitHub is the narrower document, and it is the one to trust when a tweet is larger than the folder. It says this release “provides offline inference with the core features needed for serving at scale.” Offline (you start a script, it produces an answer, it exits) is the opposite of a service (a program that stays running and answers other programs over the network). The supported model list has one line: DeepSeek-V4.1-Flash. The supported hardware list has two lines: NVIDIA SM100 GPUs, and Huawei Ascend 950 NPUs. By the evening of 20 October the repository showed about 11,000 stars, and that number moves by the hour.
Two licenses, on purpose. DeepInfer’s LICENSE is Apache 2.0. deepseek-recipe’s LICENSE is MIT. The model card says MIT for the weights. A product that ships all three is under all three. Keep each notice with the thing it covers.
What is prefill, and what is decode?
Prefill reads everything you already typed. Decode writes the reply one token at a time. They are different jobs, and DeepSeek puts them on different machines.
You send a question. That question is already text. Prefill (the pass that reads the whole input and builds the model’s working memory of it) happens before the first new token exists. It is closer to reading a stack of pages than to writing a sentence. Decode (the pass that produces the next token, then the next, using that memory) is the sentence.
The working memory has a name you will see in every speed argument: the KV cache (the stored record of what the model already computed about earlier tokens, so it does not recompute those tokens from scratch for every new word). K and V are just the two tables in that record. You can forget the letters. The point is that reading a long prompt builds a large record, and every new word has to consult it.
DeepSeek’s deployment Q&A, in the repo, describes the rooms. Prefill uses ordinary 8-GPU machines. Decode uses 64-to-128-GPU supernodes (machines tied together so tightly they act as one pool of chips). A GPU (graphics processing unit, here just a chip that does a huge number of small multiplications at once) is the worker. The smallest unit they call efficient is 64 GPUs of prefill plus 64 GPUs of decode. That is not a contradiction of the 8-GPU sentence. Sixty-four prefill GPUs is eight of those ordinary machines, standing next to one 64-GPU writing room. One hundred twenty-eight chips, before you have counted a spare.
They also split the model across those chips because it does not all have to wake up for every token. The README calls the pattern MoE EP, which unpacks as mixture of experts (a model made of many specialist slices, with only some slices used on each token) plus expert parallelism (those slices live on different chips). A load balancer (a bit of software that moves specialists around so one chip is not stuck while others wait) sits on top. Their measured imbalance, which they define as the slowest chip divided by the average, is 1.045 to 1.074 on prefill and 1.090 on decode. A perfect split would be 1. Nine percent more work on the slowest decode chip is the remainder, not a failure they hid.
What is the draft, and why does checking less of it help?
A small helper guesses several next tokens. The real model checks a prefix of that guess. DeepSeek’s twist is to check fewer tokens when it is confident, and they say the accepted length barely changes.
Decode is slow because it is a loop. One token, consult the cache, one token, consult the cache. Speculative decoding (a trick where a cheaper draft proposes several tokens, and the real model verifies them together) tries to turn the loop into a longer stride.
DeepSeek’s Q&A says the draft model generates 5 tokens at a time. A confidence head (a small extra output that scores how likely each drafted token is to be accepted) lets the scheduler verify only the prefix that looks worth it. They name this DSpark. The README’s switch is DEEPINFER_ENABLE_MTP_VARLEN=1, and DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length, default 5.
The number they publish is not “five times faster.” It is smaller and more useful. Dynamic-length verification cuts the number of draft tokens verified per step by about 30 percent, from 5.00 down to somewhere between 3.2 and 4.4. The length that actually gets accepted barely moves. At their production point, about 60 requests in flight on each chip, the Q&A computes a throughput gain of about 4 to 7 percent, and it states plus 5.4 percent at 60 sequences, plus 6.7 percent at 72, and plus 11 percent at 96. The same table shows minus 1.5 percent at 16 sequences. A batch (several requests computed together so the chip stays full) is why the sign can flip. Checking a clever prefix has overhead. Overhead wins when the chip was not busy.
Do not promote 4 percent into the launch tweet’s “high throughput.” The 4 percent is one switch, at one operating point, on their cluster. The launch tweet is the whole building.
Why can they recompute only the last page?
Some layers only look at a recent window of tokens. If that window is already known, a cache hit does not require rereading the whole conversation. DeepSeek recomputes a fixed tail instead.
Attention (the step where a token looks at other tokens to decide what matters) is the expensive look. Sliding-window attention (a version that only looks back across a fixed recent span, not the entire conversation) is abbreviated SWA in their README. The span they recompute is the most recent 128 tokens. Their hit rule, quoted from the Q&A: the longest cache hit on an input of length L is L minus 134, and they unpack 134 as a 128-token window plus 6 tokens of speculative backtracking (the extra tokens the draft may have to unwind).
The README’s default keeps 140 tokens, not 134. The comment says the retained suffix must cover the window plus the maximum backtracking length. Both numbers are theirs, and they are not the same number. The switches are DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140. The code name YOCO is not expanded in the README. The behavior is: replay a bounded suffix, so the cost of that replay stays fixed no matter how long the conversation was.
They also publish the size of the SWA record for one sequence: 20 layers times 134 tokens times 528 bytes, which is 1,415,040 bytes. A layer (one block in the stack of blocks that makes the model) is the 20. The 528 bytes per token is the format they name, MXFP8. The useful reading is not the product. It is that this particular record is about 1.4 megabytes per conversation, which is small enough to talk about in memory, not in a warehouse. The warehouse is a different record.
What is the warehouse, and why is 155 trillion not a property of the GitHub repo?
The number is one live cluster, with a disk system they did not release. Divide their own figures and you do not get the 128-GPU starter kit.
The Q&A sentence is: “A single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing).”
Take the pieces apart before repeating them.
A cluster (many machines operated as one service) is the subject of the sentence, not the repository. Tokens per day counts the reading and the writing, whatever their cache saved them from doing again, and whatever their agents resend. It is not a count of users. Divide 155 trillion by the 86,400 seconds in a day and you get about 1.79 billion tokens a second. That division is arithmetic on their sentence, not a second statistic they published. It is still the building.
Valley pricing (the cheap, off-peak price) is why US$38 per GPU-hour must not be multiplied by 24 and called a day. A valley rate pretends nothing about the expensive hours. They also do not say this is profit. Revenue (money customers paid, before the power bill, the building, and the chips) is the word they chose.
The warehouse they describe and do not ship: a KV cache system on a distributed file system they call 3FS, “roughly 100 PiB of SSD storage.” A pebibyte (1,024 tebibytes, a deliberately binary unit) is the PiB. They also say about 14.4 TB of logical cache per online GPU. Divide 100 pebibytes by 14.4 terabytes and you land in the thousands of GPUs, not at 128. The exact count wobbles because they wrote TB next to PiB, and those prefixes are not the same convention. The scale does not wobble. The starter kit is two rooms. The 100 PiB figure is a site.
Retention, from the same Q&A: keep KV cache more than 72 hours, with a time-to-live (the moment a stored page is thrown away) of 3 days for agent traffic and 7 days for chatbot traffic. They say shortening the chatbot keep to about 3 days costs 0.5 to 1 percent of hit rate. A hit (the page was already on the shelf, so you did not reread it) is the whole economic point of the warehouse. They also say the live service reaches roughly 80 percent of a perfectly balanced offline benchmark. The repo can run the offline benchmark. It does not include the balancer that gives up that 20 percent in exchange for being a service.
Their sentence for what is missing, verbatim: “The KV cache management and load balancing components of the inference system are tightly coupled to our internal infrastructure, so that code is not yet fully open-sourced.” The README’s list of what a complete service still needs is three items. One, a KV cache system. Two, VE/Prefill/Decode disaggregation (putting the vision encoder, the reader, and the writer on separate machines). A vision encoder (the piece that turns a picture into tokens) is the VE. They say most of the engine functionality for that split is already in the repo, and that one GPU is generally enough for a vision-encoder instance. Three, load balancing across the copies of prefill and of decode. The longer architecture write-up they point to is a Zhihu post. The numbers in this post come from the Q&A and the README, not from that write-up.
Can I run it on the card in today’s order, or on a PC?
Not on the evidence in the README. The chip named there is a different NVIDIA target from the workstation card Chinese firms just tried to buy, and a different target from a GeForce.
NVIDIA publishes a compute-capability table. Compute capability (a version number for what a chip can do, which the compiler turns into a name like sm_100) is the key. On that table, capability 10.0 under Data Center lists NVIDIA B200 and NVIDIA GB200. Capability 12.0 under Workstation/Consumer lists the NVIDIA RTX PRO 5500 Blackwell Workstation Edition, and also the GeForce RTX 5090 and the rest of that consumer line. The compiler’s short name drops the dot: 10.0 is sm_100, 12.0 is sm_120.
DeepSeek’s README says “NVIDIA SM100 GPUs.” It does not say RTX PRO 5500, and it does not say GeForce. Secondary write-ups read SM100 as data-center Blackwell. That reading matches NVIDIA’s table. It is still a reading. The README itself names a target, not a shopping list. Batch-invariant inference and deterministic sampling are marked NVIDIA only. Builds want Linux, GCC 13 or newer (the compiler for the C and C++ parts), Python 3.12, and CUDA 13 on NVIDIA or CANN 9.2 on Ascend. CUDA (NVIDIA’s platform for talking to its chips) and CANN (Huawei’s counterpart) are not interchangeable installers. The Ascend path is real in the README. The README gives no speed comparison between the two, and inventing one would be the failure mode of the launch posts.
The same morning, Reuters reported that ByteDance, Alibaba, and Tencent had ordered more than 3 million RTX PRO 5500 cards at about US$11,400 each, a sticker above US$35 billion, and that Nvidia had warned partners they might receive 5 to 10 percent of what they asked. A second reported figure: no more than 3 million of these cards produced globally in 2027, against more than 6 million ordered from China in total. Check the multiplication before you memorize the headline. Three million times 11,400 is 34.2 billion, so “more than 3 million” at that price does cross 35 billion. That is arithmetic on Reuters’ sources, who were not named. It is not a DeepSeek fact, and it is not a delivery. Reuters also said Nvidia cannot guarantee the orders.
Put the two stories on one table and the trap closes. The card in the order is compute capability 12.0. The NVIDIA line in DeepInfer’s README is SM100, which NVIDIA’s table aligns with capability 10.0. Open-sourcing an engine for the chip that export rules are built to restrict does not put that engine on the chip the rules still allow. It also does not mean a workstation card is useless for inference in general. It means this repository, as written on 20 October, does not claim that card.
The run command in the README is a script, python examples/offline_inference.py. The first run downloads the checkpoint (the weights file) from Hugging Face unless you point DEEPINFER_HF_MODEL_DIR at a local copy. There is no chat address in that command. If you want DeepSeek’s answers and you do not operate a data center, their API is the product that already includes the warehouse (here is how to point Claude Code at the DeepSeek API). The recipe is how a deployment speaks that API’s prompt format. It is not a reason to buy GPUs.
What did they actually speed up, and what must I not quote?
Quote the Q&A’s percentages and the README’s hardware list. Do not quote a tokens-per-second number off a chart the README table left blank.
The README says benchmarks were measured on 4 SM100 GPUs for prefill and 64 for decode, each inside one scale-up domain (a group of chips linked by the fast network, not across a slow one). The reproduction script is examples/sweep/run_sweep.sh. Under that heading the README shows two charts and no typed numbers. Other write-ups have read “above 25,000 tokens per second per GPU” off a decode chart. That figure stays in the “do not quote as typed” column until the README has typed numbers or you have run the sweep.
They do put one gap in words. The public FlashMLA decode kernel lacks one hand-tuned optimization, so it is 5 to 10 percent slower than the internal kernel, and end-to-end they put the gap at most 5 percent. That sentence is allowed. It is also a reminder that “the production stack” and “the bytes in the tarball” are not identical even where they tried to make them close.
Structured outputs, deterministic sampling, and batch-invariant mode are in the README as switches, not as the source of the 155 trillion. Structured output (forcing the answer to follow a pattern: a JSON form, a regex, or a grammar) is how a program gets a file it can parse instead of a paragraph. Deterministic sampling (same tokens in, same settings, same tokens out, by fixing the random draw) and batch invariance (the answer does not change just because a stranger’s request shared the chip) are NVIDIA-only. Useful, and not the warehouse.
| Sentence in the wild | What the sources support |
|---|---|
| They open-sourced the service that serves 155 trillion tokens a day | They open-sourced the engine. The Q&A attaches 155 trillion to a cluster, and says the cache and the load balancer are not fully open. |
| It runs on the RTX PRO 5500 | The README says NVIDIA SM100. NVIDIA’s table puts the RTX PRO 5500 at compute capability 12.0, and B200/GB200 at 10.0. |
| I can try it on a GeForce tonight | GeForce Blackwell is the same 12.0 row, not SM100. The install line is Linux, GCC 13, Python 3.12, CUDA 13. The smallest efficient unit they name is 128 data-center GPUs. |
| Speculative decoding makes it five times faster | The draft is 5 tokens long. The published gain at their busy operating point is about 4 to 7 percent, and negative when the batch is small. |
| US$38 per GPU-hour is what the cluster earns all day | They said revenue, at valley pricing. Valley means the cheap hour. They did not publish a daily average. |
| Apache 2.0 covers the weights and the recipe too | The engine is Apache 2.0. The recipe’s license file is MIT. The model card says MIT for the weights. |
Who should clone it, and what do I do if that is not me?
Clone it if you already run a room of SM100 or Ascend 950 chips and you can build a warehouse. Otherwise read the Q&A and do not rent a card for this repository.
If that is your room, the order of work is already written. Build an empty Python 3.12 environment. Export DEEPINFER_ACCELERATOR as nvidia or ascend and leave it exported. Run ./build.sh. Expect to bring your own cache, your own placement of prefill and decode, and your own balancer. Budget interconnect the way they say: roughly 400 gigabits per second per 8 GPUs. Treat the public kernel as within about 5 percent end to end of theirs, not as bit-identical. Judge a port by whether answers still match, not by a throughput chart. Pass@1 (the share of questions answered correctly on the first try) is the kind of check they point at for correctness. A passing score does not prove the cluster is fast.
If that is not your room, the valuable act is the one this post just did. Separate the file, the engine, and the building. Keep the 155 trillion on the building. Keep SM100 off the 12.0 shopping list. Keep the 4 percent on the draft switch, not on the tweet. The recipe is still worth knowing about if you call their API from your own server later: it is the piece that makes your prompts match theirs, which is a different project from making the chips match theirs.
Common questions about DeepSeek DeepInfer
Is DeepSeek DeepInfer really open source?
Yes, the engine is. DeepInfer is released under Apache 2.0. The deepseek-recipe component has an MIT license file, and the DeepSeek-V4.1-Flash model card says MIT for the weights.
Can I run DeepSeek DeepInfer on an RTX 5090 or an RTX PRO 5500?
Not on the evidence in the README. It lists NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. NVIDIA’s table puts the RTX PRO 5500 and the GeForce RTX 50 cards at compute capability 12.0, which is sm_120, not SM100.
Does the download include the KV cache system and the load balancer?
No. DeepSeek says the KV cache management and load balancing components are tightly coupled to its internal infrastructure and are not yet fully open-sourced. The release provides offline inference.
How much faster is the draft-and-check feature?
DeepSeek’s Q&A puts the gain at about 4 to 7 percent at its production operating point of about 60 sequences per chip, plus 11 percent at 96, and minus 1.5 percent at 16.
