Inside Kimi K3: How Moonshot AI Built a Sparse 2.8T Model

There is a lovely moment in Arthur C. Clarke’s fiction where a machine does something so far beyond expectation that the humans watching it stop asking how fast and start asking how. I had that moment in July 2026 when Moonshot AI shipped Kimi K3: a 2.8-trillion-parameter open-weight model, the largest ever released to the public, that somehow costs about as much to run per token as models a fraction of its size, holds a million tokens in its head at once, and still lands in the top handful of models on the planet for coding.

The obvious question, the one every developer I know asked in the same week, is: how is this even possible? A model that large should be slow, ruinously expensive, and impossible to self-host. Kimi K3 is none of those things. So this piece is my attempt to open the hood and show you the four engineering ideas that make it work, and more importantly, why each one buys the performance it does. I promise no hand-waving. Where I lean on a number, I will tell you where it came from and whether it is settled or still awaiting Moonshot’s full technical report (due 27 July 2026, so treat the benchmark figures below as provisional until then).

The Ground Floor: Why Big Models Are Supposed to Be Impossible

Let us start from the basics, because the whole trick only makes sense against the backdrop of what should happen.

A transformer, the architecture behind essentially every large language model, does two expensive things. First, it stores knowledge in its weights: the billions of numbers it learned during training. More weights means more room to memorize facts, patterns, and skills. So far, so good, bigger is smarter. Second, at every step it runs attention: each token in your input looks at every other token to decide what is relevant. That is the “T” in GPT, the mechanism from the famous 2017 paper Attention Is All You Need.

Here is the catch that makes giants impossible. In the naive design, both of those costs scale badly. If you double the parameters, every single forward pass has to touch twice as many numbers, so inference gets twice as slow and twice as expensive. And attention is worse than linear: comparing every token to every other token is a quadratic operation. Ten times the text is a hundred times the attention work. Storing all those comparisons (the “KV cache,” which I will unpack shortly) eats memory in direct proportion to how much text you are holding.

Put those together and a 2.8-trillion-parameter model with a million-token window is, on paper, a non-starter. In 16-bit precision the weights alone would need roughly 5.6 terabytes of memory, and the attention over a million tokens would melt the interconnect. You could only ever run such a thing inside a hyperscaler’s basement.

So the interesting engineering question is not “how did they train something big.” Anyone with enough GPUs can do that. The question is: how did they make something this big behave like something small enough to actually use? The answer is four ideas, each attacking one of those cost curves.

Idea One: Only Wake Up the Experts You Need

The first idea attacks the “every parameter, every time” problem. It is called a Mixture of Experts, or MoE, and the intuition is almost embarrassingly simple.

Imagine a hospital with 896 specialists. A patient walks in with a broken wrist. You do not make them consult all 896 doctors. A triage nurse glances at the problem and routes them to the orthopaedist and maybe a radiologist. Two specialists, not 896. The hospital has the collective knowledge of all 896, but any single patient only pays for the two they need.

Kimi K3 works exactly this way. Its 2.8 trillion parameters are split across 896 “expert” sub-networks. For each token flowing through the model, a small routing network picks just 16 of them to activate. The result: only about 50 billion parameters do any work on a given token, even though 2.8 trillion are sitting there available. That ratio (16 active out of 896, roughly 50B active out of 2.8T) is confirmed in Moonshot’s launch materials and independent write-ups. This is what “ultra-sparse” means, and it is why the running cost tracks the 50-billion number, not the 2.8-trillion one.

In pseudocode, the routing logic is just a top-k selection:

import torch

# router produces a score for each of the 896 experts, per token
router_scores = router(token_embedding)      # shape: (896,)

# pick the 16 highest-scoring experts for THIS token
top_vals, top_idx = torch.topk(router_scores, k=16)
gates = torch.softmax(top_vals, dim=-1)       # how much to trust each chosen expert

# run ONLY those 16, then blend their outputs by the gate weights
output = sum(gates[i] * experts[top_idx[i]](token_embedding)
             for i in range(16))

Now, where does this go wrong? With a naive router, you get routing collapse. Early in training a few experts happen to be slightly better, so the router sends everyone to them, so those experts get all the practice and improve further, so the router relies on them even more. The other 890 experts atrophy, unused, wasted capacity. It is a rich-get-richer death spiral, and it is the single hardest problem in scaling MoE.

Moonshot’s stated fix (which I will flag as their described design, since the full method awaits the technical report) is a technique they call Quantile Balancing. Instead of routing on raw score thresholds, which are brittle and hyperparameter-sensitive, tokens are assigned based on where a score falls in the ranking of all scores, its quantile. Ranking is naturally self-normalizing: if you force assignments to spread across the quantile range, every expert gets a roughly equal share of the training data no matter how the absolute scores drift. All 896 stay in the game. Alongside this, Moonshot describes a custom Per-Head Muon optimizer (Muon is a real optimizer, used already in Kimi K2, that cleans up gradient updates by orthogonalizing them; the “per-head” scheduling is K3-specific and unconfirmed until the report) and an activation function they call SiTU replacing the usual GeLU. Take those specific internal names as provisional. The principle, keep all experts busy and keep gradients stable at extreme scale, is what matters and is well established.

Why this buys performance: you get the knowledge capacity of a 2.8-trillion-parameter model at the inference cost of a 50-billion one. Capacity without the compute bill.

Idea Two: A Whiteboard Instead of a Filing Cabinet

The second idea attacks the quadratic-attention problem, and it is my favourite, because it is where Kimi K3 is genuinely, publishably novel. It is called Kimi Delta Attention (KDA), and unlike some of the internals above, it has a real peer-reviewable lineage: the Kimi Linear paper on arXiv (2510.26692), which I would encourage anyone serious about this to read in full.

Let me rebuild the problem first. Standard attention keeps a KV cache: for every token you have processed, it stores that token’s “key” and “value” vectors so future tokens can look them back up. Think of it as an ever-growing filing cabinet. Token one million arrives, and to attend properly it must riffle through all 999,999 files. The cabinet grows without bound, and the riffling is what makes attention quadratic. This is the wall that makes million-token context so painful.

Linear attention proposes something radically different: do not keep the files at all. Instead, keep a single fixed-size notebook, a matrix I will call S, and every time a token arrives, scribble its contribution into the notebook and throw the token away. The notebook never grows. To recall something later, you do not search files, you just read what the notebook currently says. Memory cost is constant. Compute per token is constant. The million-token wall vanishes.

The classic recurrence looks like this:

# S is a fixed-size memory matrix. It never grows with sequence length.
S = zeros(d, d)

for k_t, v_t in stream_of_tokens:      # k = key, v = value for this token
    S = S + outer(k_t, v_t)            # write this association into the notebook
    # to read: given a query q, the recalled value is  q @ S

Beautiful, but you have surely spotted the flaw. If you only ever add to the notebook and never erase, it fills up with a smeared average of everything you ever saw. It cannot forget, and it cannot correct a mistake. Write “the sky is green” early on and there is no clean way to overwrite it with “the sky is blue” later. This lossy smearing is exactly why early linear-attention models were fast but dumb.

KDA fixes this with two moves, and this is the heart of the whole design. The published update rule is:

$S_t = (I - \beta_t k_t k_t^\top) \cdot \mathrm{Diag}(\alpha_t) \cdot S_{t-1} + \beta_t k_t v_t^\top$

Do not let the symbols scare you; each one has a plain-English job.

The delta rule, that : $I - \beta_t k_t k_t^\top$ term, is the “correct your mistake” move. Before writing the new association for a key, it first subtracts out whatever the notebook currently predicts for that key, then writes the corrected value. It is the difference between scribbling a second, contradictory note in the margin and actually erasing the wrong answer before writing the right one. $\beta_t$ is a small gate controlling how aggressively to correct, a bit like a learning rate.)

The $\mathrm{Diag}({\alpha_t})$ term is the “forget gracefully” move, and this is KDA’s specific contribution over its predecessor, Gated DeltaNet. Older designs used one forgetting rate for an entire attention head: everything decays at the same speed. KDA uses a separate, data-dependent forgetting rate for every single channel (every feature dimension) of the memory. $\alpha_t$ is a vector, and $\mathrm{Diag}$ turns it into a diagonal matrix so each dimension decays at its own pace. Some features (say, the current programming language you are in) should persist for a long time; others (a variable name from a function you already left) can fade fast. Per-channel gating lets the model make that call feature by feature. It is the difference between one dimmer switch for the whole house and a dimmer on every bulb.

The trade-off is honest and worth stating plainly. A fixed notebook is lossy: it can never match a full filing cabinet for perfect, needle-in-a-haystack recall of one exact token from a million ago. So Kimi does not go all-in. It uses a hybrid: three KDA layers for every one full-attention layer, a 3:1 ratio. Three-quarters of the model enjoys cheap, constant-memory linear attention; one-quarter keeps the expensive, perfectly-recalling full attention for the deep exact-lookup work. The remarkable empirical result from the Kimi Linear paper is that this hybrid does not merely approximate full attention to save money, it actually outperforms pure full attention on quality across short-context, long-context, and reinforcement-learning tests, while cutting KV-cache memory by around 75% and boosting decoding throughput by up to roughly 6x at million-token lengths.

Why this buys performance: the million-token context window becomes physically runnable instead of theoretical, throughput stays high because three of every four layers barely touch memory, and, surprisingly, quality goes up rather than down. That is the rare engineering win where the cheap option is also the better one.

Idea Three: Let Deep Layers Reach Back in Time

The third idea is smaller but clever, and it solves a problem that gets worse precisely because of ideas one and two. It is called Attention Residuals (AttnRes), and here I will again flag that the exact formulation awaits the 27 July technical report; I am explaining the principle Moonshot has described.

Every deep network faces information dilution. A transformer passes data through dozens of layers in sequence, and the standard way to move information forward is the “residual connection”: each layer’s output is its computation plus the input it received, $x_{next} = x + f(x)$ . This is a game of chinese whispers. The signal from layer 3 has to survive being whispered through layers 4, 5, 6, and so on before layer 40 hears it, and it degrades along the way.

Now add sparsity. In an 896-expert MoE, a given token might hit a set of experts in the middle layers that simply are not well suited to it, muddying the signal further. The pristine linguistic or logical concept the model formed early on gets buried under intermediate processing.

AttnRes lets a deep layer reach back and attend directly to the outputs of arbitrary earlier layers, not just the one immediately below it. Instead of only hearing the last whisper, layer 40 can glance back at the original note from layer 3 whenever it needs to. In a sparse model this is especially valuable: a deep layer can effectively route around intermediate experts that fumbled a particular token and pull the clean foundational representation straight from where it was first established.

Why this buys performance: it preserves fidelity across depth, which is what lets Kimi K3 sustain coherent reasoning over the long agentic chains (hundreds of tool calls in a row) it was built for. Moonshot credits KDA and AttnRes together with a roughly 2.5x improvement in scaling efficiency over the previous K2 architecture, a figure I would still verify against the technical report before quoting it as gospel.

Idea Four: Learn to Survive Being Squeezed

The final idea is what turns all of the above from a lab curiosity into something an enterprise can actually download and run. It is Quantization-Aware Training (QAT) with a 4-bit number format called MXFP4.

Recall the terrifying number from the ground floor: 2.8 trillion parameters in standard 16-bit precision is about 5.6 terabytes. The usual response is quantization: after training, you compress the weights to fewer bits. But naive post-training quantization is like recording an orchestra in full fidelity and then playing it back through a tin can. Detail is lost, and the model gets measurably dumber.

QAT flips the order. Instead of training in high precision and squeezing afterward, you make the model aware of the squeeze while it is still learning. During training, the weights are repeatedly rounded to their compressed 4-bit form, so the model learns weights that are robust to that rounding. Here is the analogy I like: it is the difference between filming a highly detailed scene filled with fast camera movements and falling confetti, only to discover it will be streamed over a weak Wi-Fi connection—turning into a blocky, pixelated mess—versus shooting the video knowing it will be heavily compressed, and choosing clean backgrounds with smooth, steady shots. The second video comes through looking crisp and clear. QAT produces the second kind of weights.

MXFP4 is the specific format: 4-bit floating point with “microscaling,” meaning small blocks of weights share a scaling factor so the tiny 4-bit values can still cover a wide numeric range. Modern accelerators (NVIDIA’s Blackwell, AMD’s MI400 class) support it natively. The payoff is dramatic: those 5.6 terabytes collapse to roughly 1.4 terabytes, a 4x reduction, with minimal quality loss because the model was trained expecting it.

Why this buys performance: it does not make the model smarter, it makes the model exist outside a hyperscaler. 1.4 terabytes is the difference between “runnable on an on-premise GPU cluster your company can actually buy” and “impossible.” For an open-weight release, deployability is a capability.

The Deep Dive: Where the Cleverness Bites Back

No architecture is free, and I would be doing you a disservice to pretend otherwise.

The most subtle cost hides inside the MoE. Spreading 896 experts across many GPUs means that on every forward pass, tokens have to be shipped to whichever GPU holds their chosen experts, results gathered, and shipped back, twice per layer. This “dispatch and combine” traffic hammers the network fabric between GPUs. So while KDA saves you memory bandwidth, the expert routing spends networking bandwidth, and the two roughly trade off. This is why “open weight” does not mean “runs on your laptop.” A realistic Kimi K3 deployment wants a multi-node cluster with terabytes of aggregate GPU memory and fast interconnect. The dependency shifts from paying an API subscription to paying for supernode networking hardware up front.

There is also a behavioural tell that follows directly from the architecture. Because K3 leans so heavily on its full reasoning trace (“thinking mode” is always on), it is unusually sensitive to having that trace truncated. If a third-party agent harness trims the conversation history to save tokens, K3’s output can destabilize faster than models that were not built to reason continuously. Moonshot has been refreshingly candid that the model also tends toward over-eagerness, making unilateral architectural decisions on ambiguous tasks rather than asking. That is not a bug in the weights so much as a personality that emerges from optimizing hard for long-horizon autonomy. It is a reminder that architecture shapes temperament.

And a word on the benchmarks, because my editorial conscience demands it. The numbers circulating at launch (an Artificial Analysis Intelligence Index around 57, a first-place finish on the Frontend Code Arena at 1679 points ahead of Claude Fable 5, strong showings on coding-marathon tests) are genuinely impressive and come from independent evaluators and telemetry, not just Moonshot’s marketing. But they are early, some are single-run, and the full technical report is not out as I write this. Treat them as a strong provisional signal, not a settled verdict. The honest headline is narrower and more defensible than “best model”: K3 is the best open-weight model yet released, and it is competitive with the closed frontier while being self-hostable. That is remarkable enough without inflation.

Real-World Grounding

If you want to feel the difference these ideas make, watch what a developer actually does with K3 versus a conventional model. Point it at an entire codebase, hundreds of files, and ask it to refactor a subsystem. A quadratic-attention model with a small window has to be spoon-fed a few files at a time and loses the plot across the seams. K3 ingests the whole repository into its million-token window (KDA making that memory-feasible), keeps the architecture coherent in mind (AttnRes preserving the early structural understanding through forty-odd layers), pays only 50-billion-parameters-worth of compute per token (MoE), and runs on hardware the company owns rather than a metered cloud endpoint (MXFP4). Every one of the four ideas is doing a specific, load-bearing job in that single workflow. That is the whole point: they are not four unrelated tricks, they are four answers to the four ways a giant model is supposed to fail.

If you would like to actually build the attention mechanisms I have described rather than just read about them, the single best hands-on resource I know is Sebastian Raschka’s Build a Large Language Model (From Scratch), which walks you through implementing attention, KV caches, and the transformer block in plain PyTorch. Working through its attention chapter and then reading the Kimi Linear paper back to back is the fastest way I know to make KDA click from the inside. (That is an affiliate link; see the disclosure at the end.)

Reflection Questions

I do not want you to remember this piece. I want you to be able to rederive it. So sit with these:

KDA’s fixed-size memory is lossy by design, yet the hybrid model beats full attention on quality. Why might a small amount of forced forgetting actually help a model rather than only hurt it? (Hint: think about what full attention does with irrelevant tokens.)
Quantile Balancing keeps all 896 experts busy. But suppose a task genuinely only needs a handful of skills. Are we now forcing the model to use experts that are not the best fit, and if so, when does load-balancing help and when does it hurt?
Idea three (AttnRes) becomes more valuable specifically because of ideas one and two. Can you articulate the causal chain? Why does sparsity plus depth make direct layer-to-layer recall matter more than it would in a dense, shallow network?
QAT makes the model robust to 4-bit rounding. If you pushed to 2-bit or 1-bit, at what point does “the model learns to survive the squeeze” stop working, and why? What is the information-theoretic thing that eventually gives out?
If you removed one of the four ideas, which removal would be fatal to the product (not just to a benchmark), and which would merely be painful? Defend your ranking.

Practical Next Steps

If you want to move from understanding to conviction: first, read the Kimi Linear paper (arXiv 2510.26692) and find the exact equation I quoted for KDA; trace each term back to my plain-English descriptions and check that I did not cheat. Second, when the full K3 weights and technical report land (targeted for 27 July 2026), verify the two claims I flagged as provisional: the 2.5x scaling-efficiency figure and the internal names (Quantile Balancing, per-head Muon, SiTU, the AttnRes formulation). Third, implement a toy linear-attention layer with and without the delta rule on a tiny associative-recall task and watch the delta version stop smearing its memory. Ten lines of code will teach you more than ten pages of mine.

Affiliate disclosure: This article contains an affiliate link. If you purchase through it, I may earn a small commission at no extra cost to you. I only recommend resources I have used and would recommend regardless.

A note on sources and verification: the Kimi Delta Attention mechanism, the delta-rule equation, the 3:1 hybrid ratio, and the “outperforms full attention” result are drawn from the peer-reviewable Kimi Linear paper (arXiv 2510.26692). Kimi K3’s headline specifications (2.8T parameters, 16-of-896 expert activation, MXFP4 quantization, 1M-token context) are confirmed across Moonshot’s launch materials and independent coverage. Several finer internals and all benchmark figures remain provisional pending Moonshot’s full technical report; I have flagged each one inline rather than present it as settled.

About the author

Dr. Amita Kapoor is an AI researcher and educator who writes about frontier machine learning for technically curious readers. She is the founder of NePeur, which focuses on agentic AI research and development, and co-founder of Retured, which builds applied machine learning systems. Her work centres on translating deep technical research into explanations that keep their rigour while losing their intimidation, in the belief that understanding how these systems actually work is the first line of defence against both hype and fear.

Please follow and like us: