Inference path — how a model “thinks” at runtime

No training. One forward (or many for sampling). Autoregressive generation.

1. Prefill vs decode

Prefill (prompt processing)

Input: full prompt tokens t_1…t_n.
Parallel over positions (with causal mask).
Builds KV cache for all prompt positions.
Cost: attention roughly O(n² d) naive, better with kernels; memory for KV O(n d L H).

Decode (generation)

Input: one new token (or speculative multi).
Attend to all past cached K,V.
Append one K,V row.
Cost per token: O(n d) with cache (not full re-prefill).
Memory grows with n until eviction/compress.

Long chat agents = prefill of history + many decode steps, history often re-sent as text each API call (API-dependent).

2. Single decode step (detail)


1. token_id → embed → h_0

2. for layer ℓ = 1..L:

     h ← norm(h)

     q,k,v = linear maps

     append k,v to cache_ℓ

     scores = q · cache_K_ℓ^T / √d   (+ mask, RoPE)

     a = softmax(scores)

     attn_out = a · cache_V_ℓ

     h ← h + W_O(attn_out)           # residual

     h ← h + MLP(norm(h))

3. logits = unembed(norm(h_L))

4. sample or argmax → next token_id

The only sequence mixing is attention.

Everything else is per-position.

3. Where approximations can hook (Track B)

Without changing weights, an inference engine can replace:

Hook	Standard	AOC-style replacement
Store K,V for all m	Dense cache	Hybrid: local dense + far anchors
Compute a·V for far past	Full softmax over m	Lookup ŵ_j ≈ A(q) from cover
High score gap	Softmax	A0 = single top value
High r_Σ head	—	Fallback dense (gate)

Hooks require runtime control (custom attention, HF forward hooks, vLLM plugin, etc.).

Not available from pure ChatGPT web UI text alone.

4. Where I/O codecs help (Track A)

API user only controls messages:


system + history + user → API → model provider runs full dense stack

You can only:

Shorten history (summary, glyph frame, tool caps)
Cache identical prefixes (provider prompt cache)
Route easy/hard models
Prefer tools over paste

You cannot change KV layout inside a closed API.

5. Sampling and multi-step “thinking”

Temperature / top-p: distribution over next tokens.
Chain-of-thought: more decode tokens (extra compute) for intermediate text.
Tools: pause generation, run tool, inject observation tokens, continue.
Multi-agent: multiple prefills; coordinator merges.

Track A reduces tokens in history/observations.

Track B reduces cost inside each forward when you own the stack.

6. Batching, quant, GQA (practical)

Technique	Effect
GQA/MQA	Fewer KV heads → less KV memory
Quantization (4/8-bit)	Smaller weights (and sometimes KV)
Paged attention	KV in pages for many concurrent reqs
FlashAttention	Faster attention; still dense semantics

These are orthogonal to AOC. AOC changes what is stored for long past; quant changes precision/packing.

7. End-to-end agent loop (both tracks)


user event

  → Track A encode: palette + frame / Δ (optional)

  → build prompt (short)

  → model forward (Track B AOC if available else dense)

  → parse actions / tool calls

  → tools return tables (Track A discipline)

  → update frame + optional graduate to AOC anchors

  → repeat