In a nutshell: Too much pointer chasing. C# actually does much better than Java here, with its features for working with user defined value types, but it could still improve by a lot.

the jvm lacks structs and more specifically arrays of structs as a way to allocate memory. this causes extreme bloat due to object overhead as well as a ton of indirections when using large collections. the indirections destroy any semblance of locality you may have thought you had which is the absolute worst thing you can do from a performance perspective on modern processors. what people end up doing instead is making parallel arrays of primitives where there is an array for each field. this is also not ideal for locality but it's better than the alternative since there isn't a data dependency between the loads (they can all be done in parallel).

i am not that familiar with the C# runtime and i know C# has user definable value types, but i'm not sure what their limitations are.

Garbage Collection is a very consequential design decision. To free that last unused object is going to take O(total writable address space) memory bandwidth.

In addition to what others have mentioned, there's also the inability to map structures on to an area of memory. The result is that you end up using streams and other methods to accomplish the same thing, and they result in a lot of function/method overhead for reading/writing simple values to/from memory.