I've written a lot of high-performance code in C#, and in general I like it. What's really great about C# is that you can write maintainable, idiomatic, managed C# (using the GC etc), that is actually very performant for many cases. The you can dip into unsafe code (unmanaged memory, pointers etc) just for the really performance critical parts where managed code isn't fast enough (e.g. if you need to write/manage a lot of buffers).
There is more "ceremony" involved with unsafe C# code, but things like Span<T> and Memory<T> have made high-perf code so much easier in many cases.
Still, so for something that's basically all highly performance sensitive, something like Rust would be a good choice (I don't actually use Rust yet, but am planning to learn).