[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8},"actionData","errors","root",{"_106":107,"_108":109},"routes/embed.x.$handle.$name.$",{"_9":10,"_11":12,"_13":14,"_15":16,"_17":18,"_19":20,"_21":22},"handle","atiqgauri","name","visualize-gpt","path","index.html","project",{"_23":45,"_11":12,"_34":46,"_47":-5,"_48":42,"_49":50,"_51":50,"_52":53,"_54":55,"_56":57,"_58":59,"_60":33,"_61":-5,"_62":-5,"_63":37,"_64":65,"_66":67,"_68":69,"_70":71,"_72":-5,"_73":74,"_75":74,"_76":77,"_78":77,"_79":77,"_80":74,"_81":74,"_82":83,"_84":85,"_86":87,"_88":42},"file",{"_23":24,"_11":14,"_25":26,"_27":-5,"_28":29,"_30":31,"_32":33,"_34":35,"_36":37,"_38":17,"_39":40,"_41":42,"_43":-5,"_13":44},"permalink","https://www.val.town/x/atiqgauri/visualize-gpt/code/index.html","extension","html","id","cc2a8629-0458-4285-a904-aa926d14ce5c","fileId","06822f5e-68a7-11f1-9fe5-1607ee4eb77e","parentId","branchId","db2507de-f59e-485f-9119-2bcf3636ced2","branchVersion",11,"latest",true,"createdAt",["D",1781527696644],"version",1,"fileType","content","\n\n\n\n\nHow GPT predicts \"Paris\" — A guided journey\n\n\n\n\n\n\n\n\n\n\n

Latest:

\n ATTENTION — Q·K dot productplanted\n KV CACHE — O(n²) → O(n)tended\n FFN — where facts liveplanted\n

\n \n Mode · Illustrative\n Loading GPT-2…\n Mode · Real · GPT-2 Live\n

\n\n\n\n\n\n

A guided journey from input → \"Paris\" · 13 stages · one prompt · every layer

est. 2026 / GPT-Viz Annual

\n\n\n

CHAPTER ZERO — INTRODUCTION

How GPT Decides Paris

We follow a single sentence — all five tokens of it — through every layer of a GPT model. By the end, you'll feel why each step exists, not just what it does.

\n \"The capital of France is\"\n →\n Paris\n

\n\n \n

\n \n

\n Optional. The blog is fully readable without this. Flip it on to see actual GPT-2 numbers replace the illustrative ones throughout.\n

\n\n\n

\n Index ::\n 01 · Input\n 02 · Tokenize\n 03 · Embed\n 04 · Position\n 05 · Attention\n 06 · Multi-Head\n 07 · Residual\n 08 · FFN\n 09 · ×96 Stack\n 10 · Output\n 11 · Sample\n 12 · KV Cache\n 13 · Loop\n ∎ Recap\n

\n\n\n

THE WHOLE PICTURE — BEFORE WE DIVE IN

One Forward Pass, Top to Bottom

This is the entire mechanism we're about to walk through. Every stage that follows highlights its component on the map.

\n\n\n\n\n\n

\n\n\n\n

01 Where we begin

The Question Arrives

\n\n

\n You type the sentence \"The capital of France is\" and hit enter. To you, this is a half-finished thought with an obvious completion. To the model, it's not even a sentence yet — it's a sequence of bytes. The model can't read English. It can't read characters. It can only do math on numbers. Everything that follows is the journey from those bytes to a single confident guess: \"Paris\".\n

\n\n

\n Current state:\n raw string · 24 bytes\n →\n about to become: tokens\n

\n\n

Our input str

\n \"The capital of France is\"\n

\n bytes: [84, 104, 101, 32, 99, 97, 112, 105, 116, 97, 108, 32, 111, 102, 32, 70, 114, 97, 110, 99, 101, 32, 105, 115]\n

\n\n\n\n

02 First transformation

The Tokenizer Chops It Up

\n\n

\n Before anything intelligent happens, the model needs to recognize chunks it knows. A BPE tokenizer scans the string and matches it against a fixed vocabulary of ~50,000 known pieces. Our sentence breaks neatly into 5 tokens. Each gets a permanent integer ID — its address in the vocabulary. From this point on, the model never sees letters again. It only sees these IDs.\n

\n\n

\n Current state:\n 5 token IDs\n →\n about to become: 5 meaning-vectors\n

\n\n

Our 5 tokens — meet the cast of this journey

\"The\"

id 464

\"·capital\"

id 3139

\"·of\"

id 286

\"·France\"

id 4881

\"·is\"

id 318

\n The last token, \"·is\", is highlighted because it's the one that matters most — its job is to predict what comes next. Everything else exists to help \"·is\" figure that out.\n

\n\n

\n Why sub-words and not whole words? A whole-word vocab would need millions of entries — every plural, tense, name, typo. A character vocab is too short; sequences explode. BPE finds the sweet spot: common words are one token, rare words decompose into pieces. \"France\" stays one token because it's common. A rare word like \"unhopefully\" gets split into several pieces (the actual split depends on the tokenizer's learned merges).\n

\n\n

\n The leading · on most of our tokens represents a literal space. \"France\" and \" France\" are different tokens. So is \"the\" vs \"The\". Capitalization and whitespace are part of identity.\n

\n\n \n

Live BPE Tokenizer

Type any text. GPT-2's tokenizer splits it in real-time below.

\n \n \n

\n Try: antidisestablishmentarianism (one word, many tokens) · π = 3.14159 (math symbols) · print(\"hello\") (code splits differently from prose) · 你好世界 (Chinese — each character ≈ 2 tokens in GPT-2's BPE)\n

\n\n

\n The 5 token IDs above are exact, from the real GPT-2 BPE tokenizer (tiktoken). The · prefix is our display convention for a literal space — GPT-2's internal byte-pair representation uses Ġ for the same role. From here on, vector bars, attention weights, FFN neuron patterns, and logit values are plausible illustrations shaped to teach the right intuition; they are not extracted from a running model. The architecture, math, dimensions, and complexity claims are exact. Want real values? Flip the Use real GPT-2 toggle at the top of the page.\n \n Real-model mode is active. Token IDs, top-7 logits, softmax probabilities, sampled token, generated continuation, and the live tokenizer playground are actual outputs from GPT-2 small (124M params, 2019) running on your device via transformers.js + ONNX Runtime Web. The cosine-similarity playground in Stage 03 uses all-MiniLM-L6-v2. Marked sections carry a tag.\n

\n One honest caveat: GPT-2 small is a small 2019 model with limited factual recall — it often doesn't predict \"Paris\" for our prompt (it tends to generate \"now the world's largest producer of food, and…\" instead). Bigger models (GPT-3, GPT-4, modern Claude / Gemini) do reliably answer \"Paris.\" The illustrative numbers throughout reflect what large models do; the real top-7 above shows you what a tiny model actually outputs.\n

\n What stays illustrative even in real mode: the Stage 5 attention bars and Stage 3 embedding vector bars — GPT-2's ONNX export doesn't expose attention probabilities or hidden states. FFN neurons, multi-head specializations, and per-block stack labels are conceptual abstractions that don't map to single layer outputs.\n \n

\n\n\n\n

03 Giving tokens meaning

From IDs to Vectors of Meaning

\n\n

\n A token ID like 4881 is meaningless on its own. The model holds a giant lookup table — its embedding matrix, with one row per token ID. Looking up row 4881 returns a fat vector: in GPT-3, 12,288 numbers. These numbers were tuned during training to place tokens with similar meanings near each other in vector space. \"France\" ended up near \"Germany\", \"Italy\", and \"Japan\" — not because anyone told it to, but because those words appeared in similar contexts. Our 5 tokens become 5 vectors. Same shape, different meanings.\n

\n\n

\n Current state:\n 5 vectors · no position yet\n →\n about to gain: position\n

\n\n

Each token lights up its dimensions\n ○ STAYS ILLUSTRATIVE\n

\"The\"

\"·capital\"

\"·of\"

\"·France\"

\"·is\"

\n 32 DIMS SHOWN. REAL MODELS: 768 (GPT-2) — 12,288 (GPT-3). EACH ROW = ONE TOKEN'S \"MEANING FINGERPRINT\".\n

\n\n

\n The key intuition: embeddings encode meaning as direction. Mathematical operations on these vectors map to semantic operations. The famous example: king − man + woman ≈ queen. By the time training finishes, \"France\"'s vector is geometrically close to other countries — near words like \"Paris\", \"European\", \"nation\" — and far from \"banana\" or \"Tuesday\". The model didn't memorize this. It emerged from billions of training sentences.\n

\n\n

\n The embedding table for GPT-3 is 50,257 × 12,288 ≈ 617M parameters just to map IDs to vectors. Almost a full GPT-2 model, used purely as a dictionary.\n

\n\n \n

Compare two words · cosine similarity

\n Compare actual learned embeddings of any two words or phrases. Cosine similarity is +1 if identical, 0 if unrelated, −1 if opposite. Watch how concepts cluster. (Powered by all-MiniLM-L6-v2, a 25 MB sentence-embedding model — loaded on first click.)\n

\n \n vs\n \n \n

\n Try: France / Italy (countries) · King / Queen · cat / kitten · cat / airplane (unrelated) · run / walked · happy / sad (antonyms — still positive!)\n

\n\n \n

Compare many words · pairwise heatmap

\n Type up to 6 words. See every pairwise similarity as a heatmap. Watch related concepts cluster together — the diagonal is always 1.00 (self-similarity), and groups of related words form darker patches.\n

\n \n \n \n \n \n \n

\n \n

\n Try mixing categories: dog / cat / lion / hammer / saw / drill — should form two clear clusters (animals vs tools).\n

\n\n\n\n

04 Adding order

\"France is\" ≠ \"is France\"

\n\n

\n There's a problem. The next stage — attention — computes sums of these vectors. Sums don't care about order. Without position information, \"France is\" and \"is France\" would produce identical outputs. To fix this, the model adds a position vector to each token's embedding. Position 0 has its own vector. Position 1 has another. Position 4 has another. These are added directly into the token vectors. Now \"·France\" sitting at position 3 is mathematically distinguishable from \"·France\" sitting at position 0.\n

\n\n

\n Current state:\n 5 vectors · meaning + position\n →\n about to enter: attention\n

\n\n

Position Playground · drag to move \"·France\"

\n Move the slider to put \"·France\" at different positions. Watch its position vector — and the resulting sum — change. The token's meaning stays the same; only its place updates.\n

\n Position:\n

\n \n \n \n \n \n

\n ← in the prompt \"The capital of France is\", France sits at position 3\n

\"·France\" tok

+ position 3

= sum

\n Same token (top row stays put). Position vector (middle) shifts every time. Sum (bottom) is what actually enters attention.\n

\n\n

\n Concat — rejected\n Half the vector for meaning, half for position. Wastes capacity, can't directly interact.\n

\n Add — used\n Same vector encodes both. The model learns to disentangle internally — for free.\n

\n\n\n\n

05 The heart of GPT

\"·is\" Sends Out a Query

\n\n

\n Now the most important moment. Our final token \"·is\" has one job — figure out what comes next. So it does what every token here does: it sends out a query vector, essentially asking \"who in this sentence has information that's relevant to me?\" Every other token offers a key vector — its \"tag\" advertising what kind of token it is. The dot product between query and key tells us how well they match. \"·is\" is going to find out, very quickly, that two tokens matter the most: \"·capital\" (the role) and \"·France\" (the country).\n

\n\n

\n Current state:\n 5 vectors with position\n →\n about to become: 5 context-mixed vectors\n

\n\n

Step 1 — Why three vectors (Q, K, V) instead of using the token vector directly?

\n You might ask: why bother making Q and K? Couldn't \"·is\" just dot its raw vector against the others? Here's why not:\n

\n No projections\n Dot products would be symmetric: \"is\" → \"France\" would equal \"France\" → \"is\". But language is directional. \"is\" needs to look at the noun. The noun shouldn't necessarily look at \"is\".\n

\n With Q, K, V\n Each token projects through 3 different learned matrices. Now \"is\" can have a Q that points exactly where \"France\"'s K points — without forcing the reverse. Asymmetric attention.\n

X · W_Q  →  Q   # what am I asking?\nX · W_K  →  K   # what tag do I broadcast?\nX · W_V  →  V   # what content do I send if chosen?

\n\n

Step 2 — How \"·is\" scores one other token (e.g. \"·France\")

\n We dot the Q vector from \"·is\" against the K vector from \"·France\". That means: multiply each pair of dimensions, then sum them up. The result is one number — how strongly \"·is\" wants to look at \"·France\".\n

Q vector from \"·is\"

K vector from \"·France\"

Element-wise products

Click below to watch the dot product build up dimension by dimension.

—

\n \n \n

\n Geometric intuition: a dot product is large when two vectors point in the same direction. So Q and K live in the same imagined space — high dot product means \"this query matches this tag.\"\n

\n\n

Step 3 — \"·is\" scores all 5 tokens\n ○ STAYS ILLUSTRATIVE\n

\n That dot-product calculation runs against every previous token. After scaling (÷√d) and softmax, we get a probability distribution — how much \"·is\" \"looks at\" each token. Notice where the weight lands.\n

\n \n

\n The biggest weights land on \"·capital\" and \"·France\" — exactly the tokens that define what the answer is supposed to be.\n

\n\n \n

Live Attention Playground · type any sentence

\n Type a sentence below. We tokenize it, look up real MiniLM-L6 embeddings for every token, then compute real softmax(Q·Kᵀ/√d) self-attention right here in your browser. Click any token to see what it attends to — and toggle the causal mask to see how GPT-style \"no peeking forward\" changes the pattern.\n

\n \n \n

\n\n

\n \n \n scale by 1/√d\n \n \n \n apply causal mask (GPT-style)\n \n \n temperature\n \n 1.00\n \n

\n\n

Click a token to make it the query:

\n\n

\n The math here is identical to what runs inside every transformer layer of GPT-2, GPT-4, Claude, etc.: dot-product attention with optional scaling and masking. The only difference is we use MiniLM's pre-trained 384-dim embeddings as both Q and K (instead of per-layer learned projections), since browsers can't yet expose intermediate attention weights from a real LM's ONNX export.\n

\n\n

Step 4 — \"·is\" becomes an enriched version of itself

\n With those weights, \"·is\" now computes a weighted sum of the V vectors from all 5 tokens. It mostly absorbs values from \"·capital\" and \"·France\", with smaller contributions from the rest. The result replaces \"·is\"'s own vector going forward.\n

new_is_vector = 0.08·V_The + 0.34·V_capital + 0.05·V_of + 0.41·V_France + 0.12·V_is

\n \"·is\" is no longer a pure copy of the word \"is\". It now contains the influence of \"capital\" and \"France\". The same operation happens for every other token in parallel — each one mixes in whatever earlier context matters most to it. This is attention. There's nothing more to it.\n

\n\n

Why /√d before softmax?

Dot products grow with vector dimension. Without scaling, the score for the best match becomes huge — softmax collapses to ~100% on one token. Gradients die. Scaling by √d keeps the distribution healthy.

Why mask future tokens?

During training the model sees full sentences like \"The capital of France is Paris\" and learns to predict every token from the ones before it. Without a mask, the position predicting \"·is\" could attend to \"·Paris\" and trivially copy the answer. The fix: set scores for positions to the right of each query to −∞ before softmax. e^−∞=0, so they contribute nothing. At inference there's nothing to mask in our 5-token prompt — the rule simply continues to apply.

\n\n

\n Attention is O(n²·d). Going from 4k to 32k context = 64× more attention compute, the central reason long contexts get expensive.\n

\n\n\n\n

06 Different questions, in parallel

The Model Does That, But Many Times

\n\n

\n What you just saw — \"·is\" querying all other tokens — was one attention head. But one head can only learn one kind of question. So GPT runs many heads in parallel, each with its own W_Q, W_K, W_V. Different heads specialize: one looks for the proper noun, another for the role, another for syntactic glue, another for distant references. Each head produces its own enriched \"·is\" vector. Then they're concatenated and projected back to a single vector. The \"is\" that comes out has been examined from many angles at once.\n

\n\n

What each head might be paying attention to illustrative

↓ concat(head_1, ..., head_n) · W_O

\n a single \"·is\" vector, enriched by 12 different perspectives\n

\n \n

\n\n

\n Mechanistic interpretability research has identified real GPT heads that match brackets, copy the previous token, find the subject of the sentence, and dozens more. Nobody designed these behaviors — they emerged from training.\n

\n\n\n\n

07 Don't forget who you were

Add the Old \"·is\" Back In, Then Normalize

\n\n

\n Attention just replaced our \"·is\" vector with a context-mixed version. But we don't want to throw away the original — that information might still be useful for the next layer. So we add the original back in: new_is = is_before + attention_output. This is the residual connection. Then we run LayerNorm to keep the numbers in a healthy range — without it, repeated additions across 96 layers would blow up.\n

\n\n

The bookkeeping (pre-norm, used by GPT-2/3 and modern models)

x_out = x + SubLayer( LayerNorm(x) )\n        └┬┘   └────────────┬─────────────┘\n       keep             new info

\n The original 2017 Transformer wrapped the norm around the residual (post-norm). GPT-2 onward moved the norm inside the residual branch (pre-norm) — it trains much more stably at depth. Most modern open models (LLaMA, Mistral, Gemma) further replace LayerNorm with RMSNorm — same idea, no mean-centering, cheaper.\n

\n\n \n

Try it · gradient flow through 12 layers

\n Flip the toggle to see what would happen during training. The bars show how much of the gradient survives back-propagating from layer 12 down to layer 1.\n

\n \n \n

\n Without residuals, gradient magnitude collapses exponentially with depth — early layers stop learning. With residuals, the identity path keeps the gradient near 1.0 even at the bottom of a deep stack.\n

\n\n

\n Why residuals exist at all? When training a 96-layer network, gradients have to flow backward through all 96 multiplications. Each one can shrink them. The residual x + f(x) always provides a clean identity path — gradients reach early layers without dying. Without residuals, deep networks simply don't learn. This single trick unlocked everything past ~20 layers.\n

\n\n\n\n

08 Where facts live

The Feed-Forward Net Thinks About \"·is\"

\n\n

\n Attention told us who to listen to. Now the model needs to do something with that. The Feed-Forward Network is applied to each token's vector independently. For our \"·is\" — which now carries information about \"capital\" and \"France\" — the FFN acts like a giant associative memory. The pattern \"capital + France + is\" activates a specific set of neurons in the hidden layer. Those neurons collectively encode \"the answer is Paris.\" They nudge the \"·is\" vector in a direction that, in vector space, points toward the word \"Paris\".\n

\n\n

\n \"·is\" Meaning:\n \"capital-of-France-is...\"\n →\n becoming: \"...basically Paris\"\n

\n\n

The FFN expands, activates, contracts

FFN(x) = W_2 · GELU( W_1 · x + b_1 ) + b_2\n\n# x:   (768,)            ← our \"is\" vector\n# W_1: (3072, 768)       → expand to 4× wide\n# W_2: (768, 3072)       → project back down

\n\n

Hidden layer neurons (mustard = active for general context · brick = \"Paris-knowing\" neurons):

\n \n \n

\n\n

\n Where does GPT \"remember\" that Paris is the capital of France? Probing research suggests this kind of factual knowledge lives in FFN weights, not attention. Attention routes context; the FFN turns that context into \"thinking\" — including looking up facts. They're a giant key-value memory whose keys are attention outputs and whose values are concept directions.\n

\n\n

\n FFN holds ~⅔ of all parameters in a transformer. Attention gets the headlines, but the MLPs hold most of the weights and most of the knowledge.\n

\n\n\n\n

09 Depth is where intelligence lives

That Whole Thing — 96 Times

\n\n

\n Everything you just saw — attention, residual, FFN, residual — is one transformer block. GPT-3 has 96 of them stacked. Each takes the previous block's output and refines it further. The intelligence isn't in any one block — it's in the refinement across depth.\n

\n (Illustrative narrative:) By block 5, our \"·is\" vector knows it's part of a fact-completion task. Around the middle layers, the model retrieves and commits to \"Paris\" as the answer — probing studies consistently localize factual recall to middle layers, not the final ones. By the late layers, the vector has been transformed into something close to the input embedding of \" Paris\", ready for the unembedding step.\n

\n\n

What changes across the stack illustrative

\n \n

\n\n

Real depths

GPT-2 small: 12 layers, 124M params
GPT-2 XL: 48 layers, 1.5B
GPT-3: 96 layers, 175B
LLaMA-3 70B: 80 layers, 70B
LLaMA-3.1 405B: 126 layers, 405B
GPT-4 (unconfirmed leaks): ~120 layers, MoE, ~1.7T total

Why so deep?

Each block can refine the representation a little. Stacking many lets the model build up reasoning, world knowledge, and answer synthesis in increments. Depth + scale + data = the capability we call \"intelligence.\"

\n\n\n\n

10 Turning the vector back into a word

What Word Lives in This Vector?

\n\n

\n After 96 blocks of refinement, the final vector for \"·is\" is dense with meaning that essentially says \"the next token should be Paris.\" But it's still a vector — 12,288 numbers. We need to turn it back into a token. The model takes this vector and asks every word in the vocabulary: \"how close are you to this vector?\" That's done with one matrix multiplication: last_vec · W_unembed. The result is one logit per vocabulary entry — 50,257 numbers, one per token. The bigger the number, the more the model wants that token.\n

\n\n

\n Current state:\n 1 vector for \"·is\"\n →\n becoming: 50,257 scores\n

\n\n

Top 7 logits (out of 50,257)

\n Show top:\n

\n \n \n \n \n

logits[\"·Paris\"]   = 10.5   ← winning\nlogits[\"·a\"]       = 7.8\nlogits[\"·home\"]    = 7.0\nlogits[\"·one\"]     = 6.3\nlogits[\"·the\"]     = 5.9\nlogits[\"·also\"]    = 5.5\nlogits[\"·France\"]  = 4.3   ← the model \"knows\" not to repeat\n...                       (50,250 more)

\n\n

\n Weight tying: in GPT-2 and GPT-3, W_unembed is literally the transpose of the input embedding matrix W_embed. The same map that turned IDs into vectors is reused (in reverse) to score vectors against IDs — saving ~600M params at GPT-3 scale and forcing a consistent meaning space. Most modern open models (LLaMA-2/3, Mistral, PaLM, Falcon) don't tie weights — they let the unembedding learn its own mapping.\n

\n\n\n\n

11 Making it a probability

Softmax: Logits Become Probabilities

\n\n

\n Logits aren't probabilities yet — they're raw scores. We apply softmax: exponentiate everything (kills negatives, amplifies big scores), then normalize so they sum to 1. \"·Paris\" now has 87% probability. \"·a\" has 4%. The rest have crumbs. Most of the time we just pick the highest, but with a non-zero temperature, we can sample probabilistically — giving the model the option to be more creative.\n

\n\n

probs = softmax(logits / T)

\n Temperature\n \n 1.00\n \n \n \n \n

\n\n

\n Why ever sample, instead of always picking the top? Greedy decoding gets repetitive (\"Paris. Paris. Paris.\") and stuck in safe phrasings. A calibrated bit of randomness produces text that's both coherent and varied. Temperature is the dial: low = confident and deterministic, high = creative and risky.\n

\n Try sampling 100× at temperature 0.20 vs 1.50. At low T, \"Paris\" wins almost every time. At high T, a long tail of less-likely tokens starts showing up. This is the diversity-vs-coherence trade-off, visible in the histogram.\n

\n\n\n\n

12 Don't recompute what you already know

Caching the Past So We Can Move Fast

\n\n

\n We just generated \"·Paris\". The conversation isn't done — maybe we want a longer sentence: \"The capital of France is Paris, a city famous for…\". To predict the next token, the model needs to do attention again, on 6 tokens now. But here's the trick: the K and V vectors for the original 5 tokens haven't changed. They depend only on each token's own context. So we cache them. Only the new \"·Paris\" token needs fresh K and V. That's the KV cache — and it's the reason ChatGPT can stream tokens in milliseconds instead of seconds.\n

\n\n

\n KV Cache:\n empty → 5 slots filled (prefill) → +1 per new token\n

\n\n

Watch the cache grow as we generate

K cache · each slot holds one token's K across all 96 layers

V cache

\n \n \n

\n\n

\n Without cache\n Recompute K and V for every prior token at every step. Each step costs O(n²) (full attention over n tokens). Generating a sequence of n tokens: O(n³) total.\n

\n With cache\n Only the new token's K and V are computed. Each step still attends over n cached K/V, so it's O(n) per step. Generating n tokens: O(n²) total. One full power lower in n.\n

\n\n

\n Cache memory ≈ 2 · n_layers · seq_len · (n_kv_heads · d_head) · bytes. LLaMA-3 70B uses Grouped-Query Attention (8 KV heads of 128 dim each, 80 layers) — at 32k context in fp16 that's ~10 GB per user. Without GQA the same model would need ~80 GB. This is why long contexts are expensive — and why \"cached input\" tokens in APIs cost 50–90% less than fresh prefills (the cache was already built).\n

\n\n

\n Real-world optimizations: PagedAttention (vLLM) treats the cache like virtual memory. GQA / MQA (LLaMA-3, Mistral) share K/V across attention heads to shrink cache 4–8×. Continuous batching packs multiple users' caches together. These are the engineering behind low API latency.\n

\n\n\n\n

13 One token at a time, forever

And Then We Loop

\n\n

\n We have \"·Paris\". We append it. Now the prompt is \"The capital of France is Paris\". The whole journey starts again — but thanks to the KV cache, almost all the work for the first 5 tokens is skipped. The model produces the next token. Then the next. Then the next. Generation is just this loop, run until the model decides it's done (by producing an end-of-sequence token).\n

\n\n

\n Generation · edit the prompt, watch GPT continue\n ○ ILLUSTRATIVE\n ⚡ REAL · GPT-2 LIVE\n

Edit the prompt below, then click Generate.

\n \n

\n The capital of France is\n

\n \n \n \n

\n Max new tokens:\n

\n \n \n \n

\n Sampling:\n \n \n \n \n \n \n

\n Try: The recipe calls for · Dear sister, · def fibonacci(n): · In the year 2099, · any sentence fragment. GPT-2 is from 2019 — fluent at style, loose with facts.\n

# The full generation loop, in essence\ntokens = tokenize(prompt)\nkv_cache, hidden = model.prefill(tokens)         # one parallel pass, O(n²·d) attention\n\nwhile len(tokens) < max_length:\n    logits, kv_cache = model.decode(tokens[-1], kv_cache)   # O(n·d) attention per step\n    next_id = sample(softmax(logits / T))\n    tokens.append(next_id)\n    if next_id == EOS: break\n\nprint(detokenize(tokens))

\n\n

\n The model itself has no memory between API calls. When you continue a chat, the entire conversation is re-sent as the prompt every time. The model is stateless. Modern APIs (Anthropic, OpenAI) now offer prompt caching that keeps KV state alive for minutes between requests — but that's an infrastructure optimization, not memory in the model. The weights don't change; the cache is just a way to avoid re-prefilling the same tokens.\n

\n\n

\n The \"typing speed\" you see when ChatGPT streams is real, not a UI effect. Each token = one full forward pass through 96 layers in the decode phase. 50 tokens/second = each layer running in roughly 200 microseconds on a GPU.\n

\n\n\n\n

∎ Recap · All 13 stages

The Whole Journey, On One Page

\n\n

\n Pull back from the details. Here's everything you just watched — the entire mechanism that turns \"The capital of France is\" into \"Paris\":\n

\n\n

  USER PROMPT: \"The capital of France is\"\n            │\n            ▼\n   ┌──────────────────────────────────────┐\n   │ 01. TOKENIZE                         │ string → 5 IDs\n   └──────────────────────────────────────┘\n            │  [464, 3139, 286, 4881, 318]\n            ▼\n   ┌──────────────────────────────────────┐\n   │ 02. EMBED                            │ IDs → dense meaning vectors\n   └──────────────────────────────────────┘\n            │  \"France\"-vec ≈ \"Italy\"-vec, far from \"banana\"\n            ▼\n   ┌──────────────────────────────────────┐\n   │ 03. + POSITION                       │ add ordering info\n   └──────────────────────────────────────┘\n            │\n            ▼\n   ╔══════════════════════════════════════╗\n   ║   ┌─────────────────────────────┐    ║\n   ║   │ 04. ATTENTION                │    ║ \"·is\" looks at \"·capital\"\n   ║   │   Q,K,V projections         │    ║ and \"·France\" most strongly\n   ║   │   softmax(Q·Kᵀ/√d) · V      │    ║\n   ║   └────────────┬────────────────┘    ║\n   ║                ▼                     ║\n   ║   ┌─────────────────────────────┐    ║\n   ║   │ 05. MULTI-HEAD               │    ║ 12+ heads in parallel\n   ║   └────────────┬────────────────┘    ║\n   ║                ▼                     ║   × 96\n   ║   ┌─────────────────────────────┐    ║   layers\n   ║   │ 06. RESIDUAL + NORM          │    ║\n   ║   └────────────┬────────────────┘    ║\n   ║                ▼                     ║\n   ║   ┌─────────────────────────────┐    ║\n   ║   │ 07. FEED-FORWARD MLP         │    ║ activates Paris-neurons\n   ║   └────────────┬────────────────┘    ║\n   ║                ▼                     ║\n   ║   ┌─────────────────────────────┐    ║\n   ║   │ 08. RESIDUAL + NORM          │    ║\n   ║   └─────────────────────────────┘    ║\n   ╚══════════════════════════════════════╝\n            │  \"·is\" vector now points toward \"Paris\"\n            ▼\n   ┌──────────────────────────────────────┐\n   │ 09. UNEMBED → LOGITS                 │ score against every vocab token\n   └──────────────────────────────────────┘\n            │\n            ▼\n   ┌──────────────────────────────────────┐\n   │ 10. SOFTMAX → PROBS                  │ \"Paris\" = 87%\n   └──────────────────────────────────────┘\n            │\n            ▼\n   ┌──────────────────────────────────────┐\n   │ 11. SAMPLE                            │ pick a token\n   └──────────────────────────────────────┘\n            │  → \"·Paris\"\n            ▼\n   ┌──────────────────────────────────────┐\n   │ 12. STORE IN KV CACHE                │ K, V for \"·Paris\" cached\n   └──────────────────────────────────────┘\n            │\n            ▼\n   ┌──────────────────────────────────────┐\n   │ 13. LOOP — generate next             │ append, repeat\n   └──────────────────────────────────────┘

\n\n

If you remember 7 things

Embeddings turn IDs into geometric meaning.
Position is added so order survives.
Q/K/V let each token ask asymmetric questions.
Dot product + softmax = similarity → weights.
Multi-head + FFN = many views, then per-token thinking.
Residual + LayerNorm make deep stacking trainable.
KV cache drops generation one full power of n — O(n³)→O(n²).

What training actually adjusts

Architecture is fixed. Training tunes the numbers inside:

Embedding & unembedding matrices
W_Q, W_K, W_V, W_O for every head × every layer
W_1, W_2 in every FFN
LayerNorm scales

For GPT-3 that's 175B numbers, tuned by gradient descent on terabytes of text.

\n\n

\n What GPT really is: a very large, very carefully tuned next-token predictor. Everything else — answering, reasoning, coding, creativity — emerges from doing that one task well, at sufficient scale, on diverse enough data.\n

\n\n

\n One sentence to keep: \"GPT turns text into vectors, lets those vectors talk to each other and reshape themselves across many layers, then turns the final vector back into a probability over the next token — and repeats.\"\n

\n\n

\n\n\n\n\n\n\n\n\n\n\n\n\n","deleted",false,"label",[14],"c8ae0834-a977-4c73-889e-99df09eeb74f",["D",1781519848496],"deletedAt","legacyEnvVars","privacy","public","httpPrivacy","description","\"How GPT Decides Paris\"","imageUrl","https://imagedelivery.net/iHX6Ovru0O7AjmyT5yZRoA/f77753cf-f640-458f-c5e0-f34787049f00/square512?exp=1789541999&sig=4c433c95238436f2775f8eb43e94b0a5b09518ea954367d22f3807a47f05403a","userId","05b57630-c6d4-4783-9264-73ecfb47314f","pinnedFileId","0aa1c8d8-68a7-11f1-959c-1607ee4eb77e","pinPreview","pinMigrated","screenshotVersion","likeCount","orgId","e03fe3d2-a58e-4a51-846f-64e13deda473","tursoName","c8ae0834a9774c73889e99df09eeb74f","tursoHostname","c8ae0834a9774c73889e99df09eeb74f-val-town.aws-us-east-2.turso.io","tursoMeasuredAt",["D",1789464074488],"pinned","userPerm","r","valPerm","blobPerm","rw","emailPerm","sqlitePerm","projectPerm","telemetryPerm","user",{"_9":10,"_105":101},"org",{"_9":10,"_99":-5,"_100":101,"_56":57,"_102":103,"_104":42},"branch",{"_36":89,"_11":90,"_23":29,"_91":45,"_92":-5,"_93":-5,"_94":-5,"_95":-5,"_96":-5,"_97":12,"_98":10},"isOwned",14,"main","projectId","forkedBranchId","forkedBranchVersion","forkedProjectPrivacy","forkedProjectId","forkedProjectOrgId","projectName","userHandle","displayName","avatar","https://img.clerk.com/eyJ0eXBlIjoicHJveHkiLCJzcmMiOiJodHRwczovL2ltYWdlcy5jbGVyay5kZXYvb2F1dGhfZ2l0aHViL2ltZ18yaHVUVWhPWlN0WnVaVkhRc25RZG5GSmJqMnkifQ","tier","free","canRestrictVals","profileImageUrl","ctx",{"_82":-5,"_139":140,"_141":142,"_143":144,"_145":42,"_146":42,"_147":42,"_148":33,"_149":-5,"_150":-7,"_151":-7,"_152":153,"_154":42,"_155":42,"_156":-5,"_157":42,"_158":42,"_159":42,"_160":-5},"ENV",{"_110":111,"_112":113,"_114":90,"_115":116,"_117":118,"_119":120,"_121":122,"_123":124,"_125":126,"_127":128,"_129":130,"_131":132,"_133":134,"_135":136,"_137":138},"REACT_APP_SENTRY_DSN","https://586020d86f05489db63a554690acf1e9@o1398359.ingest.sentry.io/6724499","RENDER_GIT_COMMIT","e87eb04e707cf64e7e532c8e8ec395f209e17f15","RENDER_GIT_BRANCH","POSTHOG_API_TOKEN","phc_nmoyuAkStEZuUp69rYlg1DZgrIt19Gx0Y4Hwf8tGgJP","POSTHOG_API_HOST","https://wart.val.town","WEB_API_TEMPLATE","https://{val}.web.val.run","APP_URL","https://www.val.town","MODULE_URL","https://esm.town","VANITY_WEB_API_TEMPLATE","https://{label}.val.run/","DENOLS_URL","https://lsp.val.town","NODE_ENV","production","CLOUDFLARE_TURNSTILE_SITE_KEY","0x4AAAAAACtaIZJH7s9rBMZ6","GOOGLE_CLIENT_ID","875989696899-jtsqkhlbdv5jrlsk8631p7ipjg3c6o62.apps.googleusercontent.com","STRIPE_PUBLISHABLE_KEY","pk_live_51N2zvWEpZTTLipIkVEwXrUG9AT0yeOOEYxszarZT1jpsZV58Ly6wlg3SltZpHN26CUalOBmUSY7adQRQiSkCZz2500qSXNkf8Y","API_URL","https://api.val.town","orgs",[],"preferencesKeybindings","default","preferencesTheme","light","preferencesFixers","preferencesCopilot","preferencesSelectionAi","preferencesCloseBrackets","preferencesChatPanelState","preferencesDprint","preferencesDenofmt","notificationCount",0,"flagDenols","isBusinessOrgMember","businessOrgHandle","isBusinessOrgAdmin","isHigherTownieTier","navCollapsed","currentOrgId"]