What stops us is that many Universities and departments only count certain journals/conferences as “prestigious” and make both hiring and promotion decisions based on the number of publications in these venues. Even if you’re senior enough not to care about promotions, you will always have students and junior collaborators who need to run this gauntlet.

And more to the point, these prestigious conferences and journals often get the best researchers in the field to peer-review for them, because doing that work (for a prestigious venue) also holds some value in the promotion/hiring process. People like to dump on peer review, scientists more than most, but in retrospect many (some) of my papers have improved through the process of getting those people to sign off on the work.