They are loaded on top in the sense that they are contained in the upper layer(s) of the chip. So when you want to change the weights, you have to produce fewer masks for fabrication, which reduces cost and time to market.
OTOH, maybe because of this our future cyberdecks will have cartridge ports.
So I think we're looking at a spectrum here:
- Fully fixed function - i.e. Taalas
- Fixed function transformer unit (or whatever other AI architecture) with a "cartridge" for weights etc - i.e. Etched
- Flexible TPU/GPU type stuff
I think the middle of the spectrum is a bit of a dead space ATM because new models have recently been coming with significant updates to the architecture (like MoE, MTP) so by the time you have new weights you wanna load, you also want to replace the compute too. So really you probably either say "I can tolerate an old model, but I want it fast as FUCK" and go for Taalas-style, or you say "I want a near-frontier model" and you have to use flexible compute anyway.
But, caveat: this comment seems to be making me sound more knowledgeable than I actually am. Take this with a grain of salt.