Obviously there's a lot of little details you'd have to work out. Like how to make such a trusted compiler in the first place, and how to sandbox unsafe code and legacy applications that were compiled by an untrusted compiler.
If this seems like it's far-fetched or too much work, consider what lengths high-performance hardware devices like HPC network interfaces go to avoid system calls at all costs, to the point where applications talk to the hardware directly. Is that really a sustainable practice long term? And how can anyone audit the security of such hardware devices?
I'm pretty sure the discovery of Meltdown/Spectre and similar speculative execution attacks would completely wreck this model. The fix for those exploits has been to make the isolation barriers even stronger but if you don't have them at all you're wide open. If you had such an OS but then had to split it back into separate address spaces you've now lost the performance gains and just have a slower OS that is harder to develop drivers for.
That's pretty much how the Amiga worked, and that level of technology achievement is still unsurpassed today.