The underlying abstraction model of having a tree of components and re-rendering only the parts that have changed between renders doesn’t map to the hardware at all, meaning you’ll waste most of the HW performance just on maintaining the abstraction.
You’ll also get zero benefits from the third-party libraries - there’s nothing in them that can help you with stuff that matters, like minimizing amount of the GPU state transitions for example or minimizing amount of GPU/CPU syncs.
It will be scenegraphs all over again, and the graphics industry has ditched these long ago in favor of simpler models, for good reasons.
Long story short, the happy path in graphics programming is very narrow and fragile, and you typically want to structure your abstraction around it.