You wouldn't need the hardware to support arbitrary load/stores. In particular, you could get rid of (some of) the address lines...

I'm unsure if this would be much of a win.