>The jstat output also shows that CCS is at 99.21% usage, which could support this theory.
If the code cache is full, the cache sweeper will have more work to do, will run slower, and this cache is a linked list (afair), any attempt to create another C1/C2 optimised code will cause the allocator to treverse the list, try to find enough contiguous space, and fail, triggering an attempt to fragment the space. Occasionally, removing some less frequently used compiled caches.
This process isn't your normal GC process. If it runs out of memory and nothing can be removed, you're at plateau of how fast code can execute, but your JVM is consuming more CPU cycles, that means, you're losing overall performance. There is no OOM error here, it all fails and slows down silently. No exceptions, no logs, nothing but wasted CPU cycles. This is one of the worst aspects of JVM to monitor and tune. I don't know of any promethues-like metric exporters that can be used here, like in any GC activity or stack/heap metrics.
As I stated in another comment, try
`-XX:+PrintCompilation` and `-XX:+PrintInlining` and `-XX:+LogCompilation`. When this turns out to be filled, try increasing `ReservedCodeCacheSize`. This is out of your non-heap area.