Using things like stackalloc, malloc, and local-only references, It is possible to implement algorithms in C# which can run at blazing-fast speeds, sometimes hundreds of times faster than idiomatic C#. C#-style arrays would not make the cut.
The optimal thing to do would be to memcpy several KB of data at a time from your heap-based memory, into your stack-allocated memory, do your processing in the optimized inner-loop, then commit the data back to the heap. I had to do this once to increase the throughout of a cluster of image processing servers.
I often scoff at the idea that C# isn't a good choice for performance-intensive tasks.