I first started running BeOS on my PowerPC Mac clone, back when those were a thing. There was a 1.44MB floppy disk circulating which contained pretty much the full OS, along with a web browser and various utilitied and demo applications.
Inside 1.44MB. It was phenomenal.
Compared to MacOS on the same machine, raw computational benchmarks were about 40% faster, and perceptually, the gains were even better than that, with essentially zero UI lag.
So, starting from scratch can have a lot of benefits, and you don't even need to throw out that much legacy.