It is astonishing to me that the most basic of memory hacks are regarded as dark magics these days. Many things take zero extra effort to make orders of magnitude faster if you are simply aware of some basic concepts.

For instance, if you know a collection of things is going to be bounded by a reasonable quantity (e.g. less than 1mm) and that you will frequently need to perform a full linear scan of these things, you should consider using an array rather than some b-tree or other pointer-chasing type (List/Dictionary/etc). This is literally a simple "Should I use List<T> or T[MaxElements]" determination. The effects of this are very profound if many scans of these items need to be made per unit time (i.e. a game loop).

You can take this one step further by trying to make the type used in the collection a value type rather than a reference type. Depending on language/runtime/problem space, this can give you another order of magnitude uplift. And, its pretty much as simple as "public [readonly] struct" vs "public class".