A better way to think about the problem is to recognize that the APIs and drivers are providing various services that pretty much every user is going to need, and which you will now need to reimplement yourself.

Nobody needs all of Vulkan, but everyone needs quite a bit of it. Buffer allocation? Command encoding? Scheduling? Synchronization? Abstracting GPU architecture differences (and GPUs vary a lot)? Render pipeline fixed-function stages like primitive assembly, tiling, and blending? You're signing up to implement all of that - good luck!

In this view, your idea is the assertion, "I could do a better job at all that stuff than the driver developers." Maybe so! They're only human. Drivers do have bugs. But you're only human too.