Finding: Empirics distilled from Research' maths experiments

Sources:

Research'/maths/exp_data_approx/FILTER_SUMMARY_approx_output.md

Research'/maths/outputs/exp_data_7b_5050/BRUTAL_HONEST_ANALYSIS_TILL_NOW.md

Research'/maths/outputs/exp_data_7b_5050/run_*/KEY_NUMBERS.txt

Positive signals (keep)

1. Low r_Σ on some real heads

• Qwen m=256 layer0 heads: r_Σ often 4–7 (PCA 95% on A).
• Enables small r\* covers and large potential compression vs m.

1. A0 often excellent on those heads

• Piecewise-constant winner value ≈ true A when score gaps large.

1. Concrete MVP win example

• Good L0 heads: r\*≈9, ~28× vs dense V payload class metrics, very low L2 on A (filter summary).
• Not universal across layers.

1. Hybrid required

• Pure prefix manifold errors grow on mid layers / longer m.
• Exact recent + compressed past is data-forced.

Negative / limiting signals (keep)

1. Layer dependence

• Deeper heads: higher r_Σ, covering approaches m, little mem win at tight ε.

1. Model family dependence

• GPT2 extractions less compressible than Qwen L0 in reported filters.

1. Wrong object risk

• Clustering full layer attn_output (d_model) ≠ per-head d_h case; numbers look worse (brutal analysis on neo).

1. No e2e quality yet in brutal analysis

• Missing: logits, needle, PPL, generation when approx plugged in.

1. Capture fragility

• NaNs on some Qwen runs; sanitization changes data.

1. Hardware

• 8GB-class limits true 7B long-ctx collection.

Gate rule (operational finding)

if r_Σ(head) low and probe A-error < tol:

    enable AOC

else:

    dense KV

Implication for New AI Language

• Track B is conditional, not blanket.
• Marketing “always constant size” is forbidden until e2e.
• Track A remains useful regardless of engine ownership.

open_questions

• r_Σ under true generation trajectories (not only K-perturbation proxies)
• GQA models at m=8k–32k