Attention output A(q) and covering numbers

Parents:

17_attention_output_approximation_new_theorem.txt

prior_structures_core_summary.txt

18_attention_output_approx_hybrid_unification.txt

Setup

• Keys \(K=\{k_i\}_{i=1}^m\), values \(\{v_i\}\), temperature \(T\).
• Softmax weights \(a(q)=\mathrm{softmax}(K^\top q / T)\).
• Attention output (what residual uses from the head, pre W_O details aside):

\[

A(q)=\sum_i a_i(q)\, v_i = V_v\, a(q).

\]

• Query measure \(\mu\) (actual activation distribution).
• Image cloud / manifold \(M_A = A(\mathrm{supp}(\mu))\).

Core covering claim (Thm3 class)

Under Lipschitz control of \(A\) and intrinsic dimension \(r_\Sigma\) of the query (or image) support:

\[

N(M_A,\varepsilon) \le \Big(C\cdot R\cdot L_A / \varepsilon\Big)^{r_\Sigma}

\]

Independent of m when norms/effective rank behave.

So number of anchors r\* needed for ε-approx can stay small while context m grows — when r_Σ stays small.

Phase transition intuition (Thm4 class)

There is a scale m\* where approximate output covers beat dense Θ(md) storage.

Depends on ε and r_Σ. Empirics decide if real models sit in the win regime.

Oriented matroid optional layer

Keys induce hyperplane arrangement / oriented matroid; topes = chambers of constant argmax order.

Useful for theory of A0 and structure; MVP locate can use nearest prototypes without full chirotope.

Project implication

Engine language = finite set of anchors \(\{\hat w_j\}\) + locate rule ≈ language of A-space, not English.