Hacker News Reader

I am missing a mention of ROM in the article. Keeping read-only weights in RAM is rather wasteful, as ROM can be implemented more cheaply. Approaches like High Bandwidth Flash (HBF) are relevant here, and should come to market in a few years. Further optimization may be possible by tailoring for sequential access, since inference of a particular model is very predictable.

View on HN

Exactly. If we ever get models that stabilize in terms of capabilities (and arent leapfrogging each other month after month) you have a "cartridge" like thing that just plugs in with the latest model in a read only ROM thats super fast.

As far as the interconnect to the GPU/CPU - thats a different story. But with Nvidia acquiring Mellanox and Nvlink getting faster and faster I assume well get there

Modern ROM is not locked in stone. For example, Flash memory is considered a modern, advanced type of Read-Only Memory (ROM), specifically a specialized form of Electrically Erasable Programmable Read-Only Memory (EEPROM), and can be re-written.

Isnt that slower though? Maybe Im not thinking of ROM specifically (the architecture/chip) but a different form of read only cache that can be located close to the compute and is super fast

Reply on news.ycombinator.com