We frequently reference this website / guideline for a reference of maximally accessible components / web design, it's really good. Not the prettiest (thick black / yellow borders on form components and the like), but accessibility trumps design.
Good design is accessible by nature. Otherwise it’s just sparkling wank.
But yes, in general you're absolutely right, that a good designer will take into account all factors, including accessibility. But the way that "design" has evolved in practice in means that the thing we all think of as "web design[er]" is not optimizing for it.
Not everything done in the name of accesility makes it accessible to all, nor does accessibility have a necessary correlation with 'good design'.
That's not to say we should't strive for both and require accesible solutions, but let's not pretend putting lightswitches 40" from the floor or those bumpy concrete pads in grocery store parking lots are better for everyone.