The main flaw of my AI Lent concept was that it was an infrequent, high committment challenge. This is much more low stakes and fun. I will be doing it and sharing a post about my findings by the end of the year.

MLA, KDA, MoE, NoPE all attack either bytes transferred or bytes resident. MLA shrinks KV per token, KDA replaces a $O(n)$ cache with an $O(1)$ state.

Why do we care about avoiding rotary positional encoding in the first place? RoPE aliasing is one reason, but the second reason is that RoPE basically erases the compression we gained from MLAs low-rank latent, and forces materialising and caching the full rotated key.