> ... re-ordered to take into account LLM memory patterns.
If I understand you correctly, doesn't this break prefix KV caching?
If I understand you correctly, doesn't this break prefix KV caching?
This does reduce the context cache hit rate a bit, but I'm cache aware so I try to avoid repacking the early parts if I can help it. The tradeoff is 100% worth it though.