It would definitely make the engine implementation more simple and elegant, but why exactly would that be worth it? Why would that be worth all the downsides?

I think a good analogy, say that I'm implementing a database. In my database, every single time it handles a query, it always does O(n) search across every record to find matches. The implementation would be so simple and elegant!