High-Bandwidth Flash offers efficient storage for model weights

Necessity being the mother of all invention. I also thought the compute-in-memory approach was interesting re: https://mythic.ai

I didn't see cost. If cost is similar to flash drives then this could be massive. Every GPU ships with a lot of ram still but for AI inference and games you have 10TB of this stuff to stuff all your textures into and all your model weights in. For fine tuning models this would also be great if using lora or similar.

Intel really missed out, when they discontinued the Optane line, right before the RAMpocalypse.

So like.. conceptually kind of like memory mapped files on fast flash persistent storage, IIUC? Or maybe it's more like GPU-managed demand paging, caching and DMA? That could get you the capacity and better I/O characteristics.

I'm curious about how the unifying architecture is going to evolve between CPU/GPU having direct access to a singular pool of memory/storage also.

I also keep wondering when memristor technology might enter the ring, because as I understand it, it would be like moving compute into the memory, which would potentially remove the need to move the data in and out of storage as much also.

It feels like computing hardware infrastructure is fundamentally evolving.

That makes some sense. NAND Flash is massively parallel by its nature. That is rarely exposed outside the die though. You'll have that 8 bit double data rate bus and you'll learn to like it.

Now that model inference at scale is a thing though? Model weights, cached prefixes? There's a considerable demand for "slow writes, fast high bandwidth reads" memory. And every bit of storage you didn't have to use RAM for you can use for fatter KV cache instead.

I've been wondering when we'd get around to having the equivalent of "memory that runs at very GDDR6 speeds for reading, but is much slower for writing", which is exactly what you need when working with an AI model. Versus current HBM which has the same speeds writing as reading.

This feels like an exceptionally bad tradeoff. HBM is expensive due to the packaging cost and yield problems. Now you're trading the relatively inexpensive but high performance DRAM for Flash while splitting the addressable market into training and inference only hardware.

The low capacity of HBM isn't really a mistake. It's a design decision to keep the bandwidth to capacity ratio high. HBM systems with 96GB of memory tend to have around 3.5 TB/s which is a ratio of 35:1, meaning your theoretical maximum is 35 tokens of inference per second assuming you use the full storage just for parameters.

If you massively increase the capacity but keep the bandwidth the same, you just end up lowering this ratio. Your system is overall smaller, but it also has less performance.

This makes High-Bandwidth Flash an extremely niche product or the equivalent of industrially processing lampante olive oil and mixing into high quality olive oil. E.g you're spending an extreme amount of effort on making a worse product that is only marginally cheaper in absolute terms, but more expensive in terms of price to performance ratio.

Why doesn't someone bring back optane?

I hope it brings the RAM prices down!

It feels like the difference between short term and long term memory in a brain.

stick enough floppy's in parallel and you could do the same thing

Good use case for fast read, slow write.

we now only need AI companies to find a way to ruin hard disk storage and officially anything PC stops being affordable

I have fond memories of Flash 5 in early 2000s. At the time I knew BASIC and Pascal. I could figure out how to draw things, how to animate them using keyframes and motion tweening, and how to make things interactive with ActionScript, all by poking around and reading the help manual (F1) on our underpowered offline PC.