I don't think so. That might be true if we were speaking of random events (where the chances of a building being ugly or nice are more or less fixed, but we only get see the beautiful surviving old buildings).
But architecture is also defined by "schools" and has periods where public buildings follow this or that general trend -- and not all trends of all ages are equal. Some ages just have shitty design sense.
There are countries where you can see buildings from 100 and 200 and 500 even years ago, not just the exceptional ones, but rows and rows of them (e.g. the whole historical center of several European cities left mostly intact). And they still come across better than very expensive buildings by famous architects done in the same premium areas.
It's not just statistics, it's also the architectural norms that defined certain eras. An average art deco building will look better to most people's sensitivities than the most expensive (or praised by critics) brutalist or modernist architecture.
For a long time in the 20th century we had architecture design schools that produced monstrosities only critics ever cared about.
One very easy example is the ugly-ass buildings in Soviet St. Peterburg compared to the marvels built even a few decades or years before communism. Not because of lack of funds (some where expensive and were built to be enormous) but because the ideology called for ugly ass utilitarian and boxy designs.
Similarly, most celebrated Le Corbusier buildings are ugly ass affairs.
The average modern architect would rather be caught dead than design a beautiful house, with ornamentation and colors and the like -- it's about minimal or "designy" to appeal to other architects and modern critics.