https://buduroiu.com/topics/attention/Recent content in Attention on Bogdan BuduroiuHugoenAll text licensed is licensed under CC BY-NC-SA 4.0 License.Tue, 25 Aug 2026 17:28:59 +0300https://buduroiu.com/links/from-gpt2-kimi-k3/Tue, 25 Aug 2026 17:28:59 +0300https://buduroiu.com/links/from-gpt2-kimi-k3/<p>MLA, KDA, MoE, NoPE all attack either bytes transferred or bytes resident. MLA shrinks KV per token, KDA replaces a <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mi>n</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(n)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal" style="margin-right:0.02778em;">O</span><span class="mopen">(</span><span class="mord mathnormal">n</span><span class="mclose">)</span></span></span></span> cache with an <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(1)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal" style="margin-right:0.02778em;">O</span><span class="mopen">(</span><span class="mord">1</span><span class="mclose">)</span></span></span></span> state.</p>https://buduroiu.com/links/nope-no-positional-encoding/Tue, 25 Aug 2026 17:25:38 +0300https://buduroiu.com/links/nope-no-positional-encoding/<p>Why do we care about avoiding rotary positional encoding in the first place? RoPE aliasing is one reason, but the second reason is that RoPE basically erases the compression we gained from MLAs low-rank latent, and forces materialising and caching the full rotated key.</p>