Hardware Intrinsics in .NET 8

You can layout a struct (which are value types not class types in c#/dotnet) with LayoutKind attributes: https://learn.microsoft.com/en-us/dotnet/api/system.runtime....

You can also use things like Span<T> and other types for better control.

But honestly C# performs generally so well that it tends to be you only reach for these tools after you spot a performance problem and profile it, which is actually quite rare.

IME most performance problems and solutions still tend to be, "Oh this person took a naive approach here, we can use a specialised loop or data structure to fix that".

- [1] https://learn.microsoft.com/en-us/dotnet/csharp/language-ref...

There is nothing magical about the JIT, you can let it warm up, suspend the process and look at the compiled code with a debugger (prepare to be disappointed though). When working with primitive arrays, its not very different from C. Lack of value types / reinterpreting memory is a big problem for Java, but not so much for C# IIRC. Fingers crossed for project Valhalla :)

The only thing that is kind of opaque is the GC, you need to learn the basics of how the algorithm works. There are some surprising relevations, like time of GC being proportional to surviving objects, not dead ones. Not to mention the new generation of GCs which use all kinds of tricks inside their read/write barriers to trade a little bit of throughput for low latency.

I've never really saw any meaningful examples of performance optimizations from the list you provided: memory layout, cache locality, access patterns. If that would be needed, I would say that wrong language was selected for solving the problem.

In my company, we use .NET for most of our applications, with Rust being used in places where we needed raw processing speed. But to this day, I'm not 100% convinced that was good choice - I don't know how much faster is Rust in our case vs .NET written with performance as priority, and overhead of interoperability, separate team, etc. is huge.

https://go.dev/ref/mem

Just like you wouldn't allocate smart pointers in a tight C++ loop.

You can do dumb slow things in both languages, and yes, C# programmers might be less informed about these topics, but it's not a language problem.

GC is not pure downside in performance terms (at least for .NET). It actually brings some upside to the table. Deferring memory management to a batched, asynchronous background process can scale better in some applications. Your physical hardware likes to work in batches of contiguous things wherever possible.

The only case where GC burns me is by introducing latency to the application. In situations where I cannot tolerate long GC pauses, my solution is to go straight through hell - invoke GC on purpose every time my hot path iterates. This ensures that at any given moment, the amount of trash to be collected is minimal and should incur the smallest delay. This works especially well in things like game loops where you have a very obvious, well-compensated edge upon which to collect garbage.

Plenty of languages have intrisics, it is a urban myth that only C derived languages do.

https://source.dot.net/#System.Linq/System/Linq/MaxMin.cs,70

It's a little hard to read (due to the use of "ref" everywhere for raw pointer arithmetic), but that implements the use of vectorized min/max instructions to find the max of elements in an array (or any span), comparing them N at a time rather than 1 at a time.

If you don't call "new", and are careful to call non-allocating functions, and pass everything by reference, then the GC is irrelevant.

Obviously you only bother to do this at the very bottom of your flame graph in the hottest spots.

When direct control over your memory is important, you're better off using those languages. When working with idiomatic C#, you really don't think about direct memory management. The garbage collector is excellent, and in many cases, it can outperform systems like reference counting.

The primary critical tradeoff with C#'s garbage collector is memory footprint. When you directly control your memory, you can free it as soon as it's not needed. (We can typically say that the extra RAM is cheaper than developer time... Until it's not, and then "map out the layout of memory" becomes worth it.)

The secondary critical tradeoff with C#'s garbage collector is that you don't want to write a general-purpose library like SQLite in C#. (You wouldn't want to suck .Net into a Python or Java process.)

But, assuming you're writing a typical web application, server-side batch processing system, desktop application, ect, these tradeoffs often work out in your favor with C#'s garbage collector. (But, if you're running your code on 1000s of memory-constrained servers, "map out the layout of memory" may be cheaper in the long run.)

And, yes, in C# you can do direct memory management with things like unsafe code (pointers and pointer arithmetic,) and structs. In my experience, the one time I did a deep dive into those optimizations, they had no advantage over "idiomatic" C# when I hooked up a profiler.

I think there is also a buffer pool of spans in asp.net that uses spans to manage all the data coming and going from headers then tosses out the exact same headers while using one copy of it then giving that memory back to the next without having it GC'd. Web servers need a fast response time to serve more requests per a second with less resources and it's all strings all the time dealing with it. Some fancy Spans and Intrinsics baked into the framework give those gains to every call in a C# app.

I think there's a misconception about the fact that languages that have GC are slower than the ones that don't. It is universally true that you can write garbage code in any language, being either Java/C# or C/Rust.

Java a while back had the problem that when GC would run, the application would freeze for a bit due to how the GC worked. But, on the other hand, at least for Java, once everything was loaded, it would consume pretty much the same resources for 100 requests or 10000 requests because the language itself was very good at concurrency. On the other hand, C++ with boost would have it's own particular issues with concurrency and after a certain number of concurrent threads it would severely impact performance.

Those intrinsics are for people that know what they are doing, they know the architecture the code is going to run on and why using those is better than leaving to the compiler to optimize the resulting byte-code.

s/informed decisions/educated guess at most

Even a seemingly low-level C code can be transformed to such a random code by the compiler that every intuition you have is thrown out the window. Hell, even assembly can be very misleading, even expert compiler writers have to fall back to micro-benchmarks, as it can be very misleading - the longer code might just be faster.

On the heap things obviously get more interesting and the rabbit hole can get quite deep. It's different from the unmanaged (non-GC) languages but similarities apply and eventually you start to map some common ground. Most GCs at this point are multi-generational and when you are talking about The Heap you are actually generally talking about a series of heaps (which is generally not just the name, but also vaguely resembles some of the internals of the data structure as well). In .NET these are generally called Gen 0, Gen 1, Gen 2, and the LOH (which may have multiple generations as well). Gen 0 is sometimes referred to as the "nursery" or the "short-lived generation", it's generally for fresh, new objects and gets garbage collected frequently. It can get quite fragmented and very chaotic. Objects that survive for long enough get promoted to Gen 1 and then again as they survive get promoted to Gen 2 which is meant for long-lived objects. Each generation is then in turn garbage collected with less frequency. The idea being that if an object in the heap appears to be long-lived it probably is. As that assumption holds (and it may not in non-well-behaved programs) the generations also get progressively less fragmented (higher cache locality in access patterns, in best cases).

LOH stands for Large Object Heap and when objects get above a certain (sometimes configurable) memory size they get treated differently. In some cases (older GCs, GCs running in lower memory environments) the regular Gen 2 doubles as the LOH, so large objects are "assumed" to be long-lived. In more modern/more recent/more common cases the LOH has its own Gen 0/Gen 1/Gen 2 breakdown with a similar garbage collection pattern of decreasing frequency with each generation.

A lot of the cache locality/access pattern balancing acts are figuring out the "best fit" heap and trying to optimize for that. You can build very performant apps with lots of small objects that preferably never escape the Gen 0 heap. Even though garbage collection is frequent it can be very fast with simple data structures (few to no reference cycles) and if most of the objects are tossed as garbage each collection the Gen 0 remains very chaotic, but not that fragmented. Most applications by default live in some version of this state, especially early in their development. Optimizing for it can sometimes be easy, too: figure out why references are living too long; often that means simplifying objects and removing reference cycles, and relying on data structures that work that way, which can be good refactors in general (often matching naive algorithm/logical performance analysis).

In other cases, you may want to focus on pushing your objects to the long-lived Gen 2 for benefits like increased cache locality and generally less fragmented access patterns. There's all sorts of patterns to do this including the dull simple ones like just holding on to references as long as possible, to things like object pools (instead of building new objects, hold on to references to "dead"/"unused" ones and reuse/recycle them as new again).

Knowing which kind of objects should never escape Gen 0 if you can help it and which ones should move all the way to Gen 2 can be a fascinating balance to strike.

The LOH obviously has similar concerns, but they all grow in complexity because those objects are large and potentially take up a lot of memory. If you are building a lot of "Gen 0" short-lived large objects, that might cause memory pressure issues. Having some idea of why those objects are so large and also getting garbage collected so quickly can help a lot to stabilize performance. But that also isn't a reason to avoid large objects. Some types of large objects can overall improve performance as they can be ways to more easily build cache locality or improve access patterns in cases like buffers and "memory pools", especially when long-lived in an application ("Gen 2 LOH").

As for intrinsics, they mostly apply to stack space, but with some Memory<T> and Span<T> options you can "borrow check" data from heap references. GCs obviously move things around (compaction to reduce fragmentation, collection to remove garbage and promote to the next generation) so the short-lived nature of stack space is interestingly important to their safety. Most of the intrinsics work with Vector<T>, a struct (stack space) which (among other options) can be constructed from any ReadOnlySpan<T> of the right size (and layout). That ReadOnlySpan<T> could be borrowed from an object somewhere in heap space (such as an intentional memory pool in Gen 2 LOH). No "copying" happens to build the Vector<T> because the ReadOnlySpan<T> is a type-safe/GC-safe pointer at that point and the Vector<T> can't last longer than the ReadOnlySpan<T>. Cache locality of that Vector<T> and subsequent ones in an operation become based on the data structure of your objects, but especially in the case of an intentional memory pool (an array or buffered series of arrays of some form) may be however you need it to be structured (arrays of value types like integers are designed to be contiguous memory and you can arrange things how you like in arrays, just like any other language; even if the GC affects where that array lives/moves, the data within it remains contiguous in the way you write it/read it, just about as you would expect).