Of course there is open source drivers that will likely never support game-specific optimizations since for them codebase quality and overall performance is more important than few tricks that make specific games faster.
: When I say "Windows sucks", I'm just talking about the complexity inherent in dealing with a consumer operating system designed to run on arbitrary 3rd party hardware. APIs like DirectX/Direct3D provide an abstraction layer that make the complexity more manageable; and they have a benefit in that they are provided by Microsoft.
Also, the whole point of Vulkan is to move these card-specific optimizations outside of the core driver while still letting games use kernel space to interface directly with the hardware (and doubtless platform vendors will put an abstraction layer on top of that like Metal on iOS). So the whole situation I'm describing has been a huge pain for a long time, but things are starting to get better.