Maybe we go from having CPU + GPU to having CPU + GPU + FPU, where FPU = "Fast Processing Unit".
The CPU in the CPU/GPU/FPU model becomes simpler. Any time we have to choose between performance and security we choose security.
The FPU goes the other way. It is for things where speed is critical and you either don't care if others on the machine can see your data, or you are willing to jump through a few hoops in your code to protect your secrets.
For most of what most people do on their computers most of the time, performance is fine without speculative execution or branch prediction and probably even with caches that are completely flushed on every context switch. (It will probably be fine to leave branch prediction in but just reset the history on every context switch).
The FPU memory system could be designed so that there is a way to designate part of FPU memory as containing secrets. Data from that memory is automatically flushed from cache whenever there is a context switch.