Poor support for 64 bits in 2023 makes me very sad whenever I .NET
.NET in general has great support for 64-bit platforms and while it wasn't the default target for .NET Framework, it has been the default target for .NET [Core] for years.
We notably don't support single-dimensional arrays that have a 64-bit length or individual objects in general being that large. However, that also isn't necessarily a bad thing as there is a large difference between your overall application supporting more than 32-bits of memory (decently common and generally good) and individual allocations using more than 32-bits of memory (not so common and generally not good to have).
If you're creating individual allocations that are larger than 4GB, you're likely not correctly accounting for devices that only have 4-8GB of memory, are more prone to cause memory thrashing, and may end up negatively impacting other parts of the system since such allocations place special restrictions on how the GC can/should interact with the data or in the case of data that is or contains reference types it can add large numbers of additional references for the GC to track.
For such scenarios, it's often better to manage your memory differently, to support streaming/buffering where possible, and if it is the rare scenario that actually necessitates such large allocations to use things like NativeMemory to bypass the GC alongside Span or the SequenceReader/Writer APIs to work with smaller chunks of data. This also allows the data to be thought about differently since the algorithm you'd use to work with small to medium sized data is not necessarily the same algorithm you'd want to use when working with large to huge data. The time cost required to process multiple GB of memory is significantly different after all, as is the cache locality and other aspects.
At this point even my phone has 12GB of memory. Although I am more concerned about deep learning. Streaming tensors doesn't make much sense.
The other annoying bit is memory-mapped files.
That being said, it is largely in the same boat where if you're working with more than 2GB of memory in a single allocation, you're not necessarily going to have a good time. Even if your phone has 12GB of memory, you in practice get a much smaller portion of that available for actual usage and you don't want to cap out the available RAM either since that will cause thrashing and other issues.
75-80% of available physical RAM is about what you want to cap out at, and that's if you're the only app really using any resources.
MemoryMapped files notably support 64-bit values today and have since they were introduced. As have streams in general.
Streaming tensors can make sense and is necessary with extremely large models. The CPU cannot handle all data at once and the cost of loading everything from disk is often expensive, so you'll want to balance loading in data you need in the short term with data that isn't needed anymore or will only be needed in the long term.
Streaming or otherwise chunking your data can likewise help with parallelization and while there are some domains where chunking isn't possible, streaming always is. It just comes down to data size to determine if its beneficial or not.