SD card and SSD's have a latency on the order of 500us, while even a Raspberry Pi 3 has a memory latency[1] of around 200ns. That's three orders of magnitude difference in latency, which would destroy performance.
In addition, memory bandwidth is still at least an order of magnitude higher.