MLA, KDA, MoE, NoPE all attack either bytes transferred or bytes resident. MLA shrinks KV per token, KDA replaces a $O(n)$ cache with an $O(1)$ state.

Why do we care about avoiding rotary positional encoding in the first place? RoPE aliasing is one reason, but the second reason is that RoPE basically erases the compression we gained from MLAs low-rank latent, and forces materialising and caching the full rotated key.

I’m only now starting to dive deep into Atproto and the design decisions being made. The protocol seems very flexible for different social media applications, and well designed to the point that the barrier to entry, whether from the hardware or ability POV, is not daunting. Finding this made me happy recently.