You wouldn't need the hardware to support arbitrary load/stores. In particular, you could get rid of (some of) the address lines...
I'm unsure if this would be much of a win.
well, the data is way smaller than the model (at least the current trend), and you probably still need random access for the weights of the model. I am not sure if it a gain worth to pursue.