Just immagine having to scan through 100 articles every day looking for new ideas. You find something that sounds interesting from somebody you do not know personally (happens to me all the time). The topic is of your interest, but you are not necessarily an expert of that particular subfield (because you are an observer, a "user" of models, not a "developer"). Judging its quality would take away one day of your work. Clearly, if there is a comment from a famous established professor saying that the work is wrong, you will happily forget about it.
It is also true that a famous professor can backstab you when reviewing your paper. But, first of all, your work is already on the arXiv, so everybody already had a chance to form their own opinion. Secondly, in a peer review process the reviewer cannot arbitrarily reject papers. It does not work like that. He/she has to provide good motivations for his/her recommendations. Reviewers can also be challenged to the editors, who can ask for second or third opinions.
Finally, as others commented, to truly disseminate your work you need to go out and engage the community, giving seminars and talks. I couldn't agree more with this. My issue with that, and I talk as a privileged because I work in one of the top Universities in the USA, is that going to conferences, giving seminars and so on, is way easier if you come from one of the top places. You have funds for traveling, you had a lot of chances to network with the right people (when they visited your institution, for example) and so on. It is much more difficult if you come from lesser known groups or from abroad.
The system as it is now already strongly favors people working in the top research Universities in the USA and Canada.
The arXiv is great because it puts everybody at the same level. It ensures that the best ideas have a chance to come out, independently from their origin. I wouldn't want the scientific discussion to be dominated by few loud voices.