22,580 GPT-2s Fit Inside Kimi K3. Scale Is the Boring Part.

Twenty-two thousand five hundred and eighty. That’s how many GPT-2s you could build from the parameters in Kimi K3, if parameters were bricks.

GPT-2 arrived in 2019 at 124 million parameters (OpenAI). K3 arrived in 2026 with 2.8 trillion (Kimi Team). Seven years, four and a half orders of magnitude. And yet if you put their block diagrams next to each other, you would struggle to tell them apart: embed the tokens, run them through a stack of blocks, each block doing attention and then a feed-forward bit, add everything back into a running total, turn the last hidden state into logits. Same skeleton.

So is it all just scale?

Not quite, and the difference is more interesting than the size. Nearly every architectural change between those two models is about memory: what the model holds on to while it reads, how it writes into that memory, what it throws away when the memory fills up, and who decides. Scale bought the room. Architecture decides what goes in it.

Here’s the path, one idea at a time.

The 2019 model you can still read in an afternoon

GPT-2 is a decoder-only transformer, which is a fancy way of saying it reads left to right and writes one token at a time. Strip away the plumbing and a block is four lines:

def forward(self, x):
    x = x + self.attn(self.ln_1(x))   # look back at everything so far
    x = x + self.mlp(self.ln_2(x))    # think about it privately
    return x

Those two x = x + ... lines are the residual stream: a running total that every layer reads from and adds to. Remember it, because the last architecture in this post finally picks a fight with it.

The small GPT-2 has twelve of these blocks, twelve attention heads, 768 dimensions and about 50,000 possible tokens. That’s the 124 million.

Now watch what happens when it writes. The model produces a hidden state for every position in the input, but when it’s generating, it only ever uses the logits at the very last position, because that’s the one that picks the next token. Everything else it just computed is scaffolding.

Worse, without help, appending that new token and running the whole thing again would recompute every previous token’s keys and values from scratch, and they haven’t changed. So you keep them. That’s the key-value cache, and it’s the first memory in this story: a growing pile of vectors, one set per token, that the model consults every time it writes a word.

It works beautifully, and it has one property that shapes everything after it. The pile never stops growing.

That property is the whole reason the rest of this post exists, so it’s worth feeling rather than reading. Drag the context slider below and watch what happens to the batch: every doubling of context halves the number of conversations a card can hold, because the weights don’t move and the cache does.¹¹The defaults here are roughly K3’s shape: 8 key-value heads at 128 dimensions each. Grouped-query attention is the reason that first number is 8 rather than the 64 you might expect, and it is the single cheapest trick on this page.

2020: throw the cache away

Softmax attention makes a score for every pair of query and key, then normalises across them. The exponential is applied after the query meets the key, which is exactly what couples them: you cannot simplify the expression, so you compute all N² pairs.

Linear attention asks what happens if you apply the nonlinearity first, separately, to the query and the key. Katharopoulos and colleagues used elu plus one, choosing elu over relu so the gradient doesn’t die when the input is negative (Katharopoulos et al.). Once the feature map is applied separately, matrix multiplication is associative again, so instead of (QKᵀ)V you can compute Q(KᵀV). The keys and values collapse into one fixed matrix, and reading is a single multiply:

k = F.elu(k) + 1
q = F.elu(q) + 1
S = S + k.transpose(-1, -2) @ v   # fold this token into the state
z = z + k                         # and into the normaliser
o = (q @ S) / (q @ z)             # read it back

The cache stops growing. What you carry between tokens is a d by d matrix per head, the same size at token ten and token ten million.

The paper reports up to 4,000 times faster autoregressive generation on very long sequences, and that number is real but worth reading carefully.

One line in it sent me down a rabbit hole. The paper says the cost per timestep for a transformer “scales with the square of the current sequence length”. That isn’t right if you keep a key-value cache: each new token attends to the N tokens before it, so the step is linear and the whole generation is quadratic. Then I checked the date. The paper is from June 2020. FlashAttention landed in May 2022, two years later, and at the time plenty of reference implementations really did materialise the full N by N matrix and recompute history. The claim describes the code people were actually running, which is a good reminder that papers are written inside a moment, not above it.

There is a real cost, though. The exponential in softmax is an extremely expressive similarity function, and elu plus one is a cheaper stand-in. Attention, at heart, is three steps: make the scores non-negative, divide by their sum, take a weighted average of the values. Linear attention keeps all three. It just uses a blunter instrument for the first one.

A notebook with no eraser

Here’s the catch with a fixed-size state, and it’s a good one.

Writing is addition: S = S + kᵀv. The state has no slots. Token 5 and token 500 land in the same matrix, on top of each other. Reading works when keys point in different directions, and the moment two keys overlap, each one drags a bit of the other’s value along with it.

Schlag and colleagues put it plainly in the paper that connected linear attention to 1990s fast weight programmers: adding new associations to a memory of finite size “will reach a limit”, and to avoid interference on retrieval, the keys need to be orthogonal (Schlag et al.). Once the sequence is much longer than the state is wide, which is precisely when you wanted linear attention in the first place, you are over capacity and everything is smeared into everything else.

Their fix is the delta rule, and it’s the kind of idea that seems obvious once you’ve seen it. Before writing, read. Ask the state what it currently thinks is stored at this key, subtract that from the value you meant to write, and write only the difference:

v_old = k @ S                       # what's already here?
u = beta * (v - v_old)              # only what's actually new
S = S + k.transpose(-1, -2) @ u     # same write, smaller correction

That beta is a learned write strength, between zero and one: how firmly to commit this particular fact.

To see why it helps, take the smallest example that shows anything. Two facts, two keys that point 0.7 of the way towards each other, each storing the value 1.00. Ask for them back.

Pure addition gives you 1.70 for both, because each key picks up seven tenths of the other one’s value. The delta rule gives back exactly 1.00 for the fact it wrote last, and cuts the error on the older one from 70% to 21%. It doesn’t make interference vanish. It stops it from compounding.

The sliders are worth a minute. Turning β down is what a model does when it isn’t sure, and you can watch the correction stop short.²²β is learned per token per head, so a real model is making this call constantly rather than once. The figure holds it still so you can see what the dial does. And the error left on the older fact is worst around half overlap, not at full overlap, which surprised me: a key that points almost exactly the same way as another is also a key the correction can reach.

Making it fast enough to actually train

This is the part that cost me an afternoon. Seven hours, if we’re counting.

The delta rule has a nasty property: every write depends on the state left by the write before it. Read, subtract, write, repeat, one token at a time. That’s fine when you’re generating (you’re stuck going one token at a time anyway), and it’s ruinous when you’re training, where the whole point is to process thousands of tokens at once on hardware that wants big matrix multiplications.

The trick is to stop thinking token by token and start thinking chunk by chunk. Inside a chunk, do ordinary masked attention: scores first, the familiar order. Between chunks, do the recurrent thing: fold everything into the state and read it back with one multiply.

for i in range(t // C):
    q_c, k_c, v_c = q[:, :, i*C:(i+1)*C], k[:, :, i*C:(i+1)*C], v[:, :, i*C:(i+1)*C]
    o_prev = q_c @ S                              # everything before this chunk
    o_curr = (q_c @ k_c.transpose(-1, -2)).tril() @ v_c   # within this chunk
    S = S + k_c.transpose(-1, -2) @ v_c
    outs.append(o_prev + o_curr)

The delta rule makes this harder than it looks, because every correction needs the state as it stood at that token. Yang and colleagues get around it by rewriting the recurrence so that a chunk’s worth of corrections can be solved in one go, using products of Householder matrices, which is what turns DeltaNet from a nice idea into something you can train at scale (Yang et al.).

What I find genuinely lovely is that the chunk size is a dial between the two worlds. Set the chunk to the whole sequence and you have written ordinary quadratic attention. Set it to one and you have plain recurrence. Everything in between is a trade.

Two terms, and only one of them cares about the chunk size. The state work is flat. The little attention matrix inside each chunk grows with the chunk, and at full length it is the quadratic cost everyone was trying to escape, about 43 times the arithmetic of a chunk of 64.

So why isn’t everyone running a chunk size of one, which is cheapest of all? Because FLOPs are not time. A GPU would rather do more arithmetic in a shape its matrix-multiply units recognise than less arithmetic in a shape they don’t, which is why the kernels land on 64 or 128 and stay there.

Forgetting on purpose

The delta rule can replace a fact when it has a key to look it up by. What it cannot do is clear the desk.

Say the conversation changes topic. Nothing in the delta rule frees space in general; it only overwrites when a new key happens to collide with an old one. Meanwhile the state is finite and still filling up.

The state-space model line of work had the opposite tool. Mamba-2 and its relatives decay the whole state a little at every step (Dao and Gu):

S = alpha * S + k.transpose(-1, -2) @ v   # forget a bit, then write

That keeps the state from filling up, and it forgets like a flood rather than an edit: every association fades at the same rate, whether it was the load-bearing fact in the document or a stray preposition.

You can see where this is going. Gated DeltaNet puts the two together, and the reason it works is stated neatly in the paper: gating erases memory quickly, the delta rule edits it precisely, and the two are complementary (Yang et al.). One parameter controls the blend. At one, you have the pure delta rule. At zero, you have wiped the board. In between, the model chooses how hard to forget, per token, from the token itself.

That’s the pattern worth noticing, because it happens again and again from here: the fix is never more memory. It’s a better-informed decision about what to drop.

One dial per channel

Gated DeltaNet uses one decay value per head. Every channel in that head forgets at the same rate, which is a bit like having one volume knob for an entire orchestra.

Kimi Linear’s contribution, Kimi Delta Attention, is to give each feature dimension its own forgetting rate (Kimi Team). Finer control over the same underlying idea. It also interleaves these layers with ordinary full-attention layers in a fixed three to one ratio, so the model always has a few places where it can look at the raw context rather than its compressed memory of it.

The headline claim is the one that made people sit up: under like-for-like comparison it beats full attention, rather than merely getting close to it for less money. The efficiency numbers deserve a careful read, though. The paper reports up to 75% less key-value cache and up to six times the decoding throughput at a million tokens of context, and it says plainly that the 6.3 times figure is a per-token latency comparison, while the measured end-to-end speedup at that context length is 2.3 times. Both numbers are in the paper. Only one of them is a stopwatch.

Kimi K3, layer by layer

Which brings us to the 2.8 trillion.

K3’s backbone is 93 layers: three Kimi Delta Attention layers, then one gated full-attention layer, repeated, with one extra global layer at the very end so the last word the model says is informed by everything, not by a summary of everything. That works out to 69 fixed-memory layers and 24 full-attention ones (Kimi Team).

The width is sparse. Of 2.8 trillion parameters, any single token touches about 104 billion, because the feed-forward layers are mixtures of experts: 896 routed experts of which 16 fire per token, plus two shared ones that see everything. That is a sparsity of 56 to one. The trick that makes 896 experts affordable is that the routed ones work in a narrow latent space, while the shared experts keep the full width.

Extreme sparsity is unstable, so the report describes two guards: a normalisation step before the up-projection, and a new activation called SiTU, for Sigmoid Tanh Unit, which is bounded rather than free to explode. Same shape as the usual gated unit near zero, a ceiling far from it.

gate, up = x.chunk(2, dim=-1)
situ = self.beta * torch.tanh(gate / self.beta) * torch.sigmoid(gate)
return situ * up

The attention layers get a gate too: the gated part of gated MLA is a full-rank, input-dependent gate on the output, deciding how much of what attention just retrieved is allowed into the residual stream. Same theme again. Retrieval is cheap; deciding what to keep is the hard part.

Attention, pointed at depth

The last change is my favourite, because it goes after the oldest line in the file.

Remember x = x + attn(x). Every layer reads one running total and adds to it. That total is the only channel through which layer 3 can tell layer 80 anything, and everything in it is weighted equally. The K3 report describes this as compressing all prior information into a single state over depth, a bottleneck it compares to an RNN over time.

Which is a pointed comparison, because we know what fixed that for sequences. Attention did.

So Attention Residuals do it for depth. Each layer gets a small learned query, the earlier layers’ outputs act as keys and values, the scores go through a softmax, and the layer reads a weighted mixture of the past instead of the flat sum. Layers whose outputs are simply large don’t get to dominate, because the keys are normalised first.

Doing that at every layer over every earlier layer costs real memory, so K3 does it blockwise: 93 layers cut into eight blocks of twelve, each block summarised by a sum, with attention across the blocks. Depth-wise retrieval at a coarser grain, for a fraction of the bookkeeping.

Notice what the two retrieval mechanisms have in common. The delta layers carry a fixed-size memory over the sequence and must eventually drop something; full attention layers go back to the raw context; attention residuals go back through the depth of the network. Three different ways of saying: don’t rely on the summary, go and look.

So what actually changed?

Not the skeleton. Embeddings, attention, a feed-forward layer, a residual stream, a head: GPT-2 would recognise every part of K3, and a 2019 reader dropped into the code would be fine after an hour.

What changed is that every one of these steps is a better answer to the same question. A fixed-capacity memory needs an eviction policy, because addition alone interferes once the memory is full. The delta rule gives you a targeted write. Gating gives you a general erase. Per-channel gating gives you a finer erase. Mixtures of experts give you width you only pay for when you use it. Attention residuals give you a second opinion about your own past.

Every one of them is learned selection: the model deciding what to keep and what to let go, rather than being told. And every time the question comes up, the most effective answer has been attention, pointed at whatever is being forgotten this time. Sequence, then memory, and now depth.

Scale bought the room. The interesting seven years were spent deciding what to put in it.

Sources

Every architectural claim above comes from one of these, and I’ve tried to keep the difference between what a paper reports and what I think about it visible in the prose. The figures are mine, drawn from the numbers in the text; the toy example and the arithmetic in them are described in each figure’s source line.

Models and documentation

• OpenAI, GPT-2 model card, for the released sizes
• Andrej Karpathy, nanoGPT, the GPT-2 implementation the code sketches follow

Research

• Katharopoulos et al., Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention, 2020
• Schlag et al., Linear Transformers Are Secretly Fast Weight Programmers, 2021
• Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022
• DeepSeek-AI, DeepSeek-V2, 2024, which introduced multi-head latent attention
• Dao and Gu, Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, 2024
• Yang et al., Parallelizing Linear Transformers with the Delta Rule over Sequence Length, 2024
• Yang et al., Gated Delta Networks: Improving Mamba2 with Delta Rule, 2024
• Kimi Team, Kimi Linear: An Expressive, Efficient Attention Architecture, 2025
• Kimi Team, Kimi K3: Open Frontier Intelligence, 2026