Track B — Attention Output Cover (AOC)

Also called: Manifold KV, output-manifold cache, O_new (research).

Status: Forced architectural target from maths 17/18 + empirics. MVP = hybrid anchors; full matroid optional.

1. Problem restatement

Standard cache stores keys and values for all m positions.

What residual needs from attention is:


A(q) = softmax(K^T q / T) @ V

If the map q ↦ A(q) has low intrinsic dimension on real queries, a small ε-net of A values can approximate the contribution without storing all K,V.

2. Object

Per head (or per layer aggregate — product choice):


O_AOC = {

  protos[1..r*]:    vectors for locate (often key-space or residual-space)

  anchors[1..r*]:   ŵ_j ∈ ℝ^{d_v}   // approx A outputs

  local_KV:         last W positions exact K,V

  r_sigma:          estimated intrinsic dim

  mode:             soft_anchor | A0 | dense_fallback

  stats:            error probes, domain tag

}

3. Algorithms

Build (prefill / graduate chunk)


1. Collect K,V (or true attn outputs A) on a chunk

2. Estimate r_Σ (PCA 95% on A-cloud or proxy)

3. If r_Σ > threshold: mark DENSE; keep full KV; return

4. Choose r* ≈ c · r_Σ (e.g. c=3..5) or elbow for target ε

5. Cluster A-cloud → anchors ŵ_j

6. Farthest/kmeans on K → protos for locate

7. Keep only last W tokens in local_KV

8. Discard bulk K,V for this head

Query


A_local = standard_attention(q, local_KV)   # may be 0 if empty

j* = argmin_j dist(q, protos[j])            # or cos sim

if mode A0 and score_gap high: A_far = V_of_winner

else: A_far = anchors[j*]

return A_local + A_far   # residual add after W_O as usual

Online

New tokens enter local_KV.
When local full: graduate oldest segment → rebuild/refresh anchors (barycentric or light recluster).
Paid novelty: only grow r* when new A samples exceed ε from all anchors (research: chamber splits / Ξ^pay).

4. Modes

Mode	When	Behavior
soft_anchor	Default low r_Σ	Nearest (or soft-k) anchor
A0	Large top-2 score gap	Single top value — often excellent (math Thm1 + data)
dense	High r_Σ or probe fail	Full KV

r_Σ gate is mandatory (filter summary): deep layers / some models may not win.

5. What this is / is not

Is	Is not
Change of stored activation structure	A new written human language
Frozen inference optimization	Free exact infinite context
Supported by covering theory when r_Σ low	Proven e2e on all 7B long tasks yet

6. Implementation pointers in monorepo

Artifact	Path
In-tree MVP (Track B)	`new_ai_language/aoc/` — `hybrid_aoc.py`, `run_mvp.py`
Harvested practical spec	`Research'/maths/outputs/HARVESTED_COHERENT_SPEC.md`
Theorems	`Research'/maths/outputs/17_attention_output_approximation_new_theorem.txt`
Hybrid plan	`Research'/maths/outputs/18_*`
Code seeds	`Research'/maths/outputs/harvested_practical_approx.py`, `math_kv/attention_output_approx.py`
Empirics	`Research'/maths/exp_data_approx/`, `FILTER_SUMMARY_approx_output.md`
Brutal honesty	`Research'/maths/outputs/exp_data_7b_5050/BRUTAL_HONEST_ANALYSIS_TILL_NOW.md`

Run (from monorepo root)


python -m new_ai_language.aoc.run_mvp --synthetic --m 256 --d 64

Backend is NumPy (no torch required for synthetic). Real .pt raw_kv needs torch.

7. Minimal success metrics

Size: r\* · d_v + W · d vs m · d (per head).
A-error: mean/max ||A_hat − A|| on held queries.
Gate rate: fraction heads in AOC vs dense.
E2E (required before ship): needle / PPL / task score vs dense.

8. Open risks

Capture NaNs on some HF paths
Layer-level vs per-head object mismatch
RoPE / GQA composition
Throughput vs FlashAttention dense path
r_Σ growth with true long multi-domain context