https://buduroiu.com/topics/transfomers/Recent content in Transfomers on Bogdan BuduroiuHugoenAll text licensed is licensed under CC BY-NC-SA 4.0 License.Tue, 25 Aug 2026 17:25:38 +0300https://buduroiu.com/links/nope-no-positional-encoding/Tue, 25 Aug 2026 17:25:38 +0300https://buduroiu.com/links/nope-no-positional-encoding/<p>Why do we care about avoiding rotary positional encoding in the first place? RoPE aliasing is one reason, but the second reason is that RoPE basically erases the compression we gained from MLAs low-rank latent, and forces materialising and caching the full rotated key.</p>