How modern AI (transformers) work — architecture

Audience: agents building representation systems. No marketing.

1. Big picture

A decoder-only transformer (GPT-class) is a stack of layers that repeatedly:

1. Read a sequence of vectors (residual stream).
2. Mix information across positions (attention).
3. Mix information within each position (MLP).
4. Write back into the residual stream.

Weights are fixed at inference. “Thinking” is forward computation on activations.

2. From text to vectors

characters/bytes

    → tokenizer (BPE / SentencePiece / …)

    → integer token ids t_1, …, t_n

    → embedding matrix E  (vocab × d_model)

    → x_i = E[t_i] + position_encoding(i)   (RoPE often applied inside attention, not always additive)

Why this matters for “language”

• The model never sees English spelling after tokenize.
• BPE is a compression of text for storage in a table, not a semantic language.
• One human concept can be many tokens; one token can be a common fragment (e.g. "ing").
• Token count ≠ information content in any deep sense — it is interface cost.

3. Residual stream

After embedding, each position holds a vector h ∈ ℝ^{d_model}.

Every sublayer does roughly:

h ← h + Sublayer(Norm(h))     # pre-norm style (common)

So the residual stream is a continuous working tape.

Mechanistic interp often treats it as the place where “features” live.

Implication: if you care about native representation, h (and things computed from it) matter more than the string that produced it.

4. Attention (one head, simplified)

From residual (after linear maps):

Q = h W_Q      # queries   shape [n, d_h]

K = h W_K      # keys

V = h W_V      # values



scores_ij = (q_i · k_j) / √d_h     # or RoPE-modified dots

a_i     = softmax(scores_i,:)

out_i   = Σ_j a_ij v_j             # attention OUTPUT A for that query position

Multi-head: several heads in parallel, concat, project with W_O, add to residual.

Objects

Softmax geometry (intuition)

• High temperature / flat scores → average of many values.
• Low temperature / large gap → nearly winner-take-all (one value dominates).

Your research formalizes this with gaps Δ(q) and A0 ≈ v_{argmax}.

5. MLP / FFN

After attention:

h ← h + MLP(Norm(h))

MLP is position-wise; it does not mix sequence positions.

Often largest parameter block. For long-context memory, attention/KV dominates runtime memory, not MLP weights (weights shared).

6. Stacking layers

L layers: each has its own W_Q,K,V,O and MLP.

Deep networks: early layers often more “local/syntactic”; later more “semantic” — empirical tendency, not a law.

Your filter: early layers sometimes lower r_Σ on A → better for output covers.

7. KV cache (inference architecture fact)

When generating token n+1 autoregressively, past K and V for positions 1…n are reused:

cache_K[layer, head] : [m, d_h]

cache_V[layer, head] : [m, d_h]

New token only computes new q and one new k,v row, then attends to full cache.

Memory ∝ m · d_h · layers · heads · bytes (with GQA, key/value heads may be shared).

This is the engine tax of long context.

Prompt token tax is related (longer prompt → larger m) but the representation stored is continuous K,V — not English.

8. Training vs inference (gradients)

No-retrain constraint means:

• You may change prompts, tools, caches, routing, how you store K/V or approximate A.
• You may not freely learn new soft embeddings / new attention ops unless marked needs-train.

Activation steering / soft prompts sit on the boundary (need stored vectors, often trained).

9. What “language” means at three levels

New AI Language project cares about all three, but must not confuse them.

10. Related docs

• 03_inference_path.md — step-by-step generate loop
• 04_why_text_is_hostile.md — cost and mismatch
• maths/ — formal A(q) covering results