You could argue that publishers only ever needed reviewers - and all the administration baggage that you mention that comes with it - because they had to choose what to compile into each paper issue that would be mailed to subscribers. If we remove the concept of "issues" and just have everyone self-publish on arXiv, a lot of the value you mention regarding journals is no longer needed.
Of course, everyone publishing on arXiv has downsides. It's no longer easy to just read Nature/Science/Physical Review Letters/etc. to find the best research in the field - some other mechanism will be needed to show scientists the best papers without them spending huge portions of their time reading - but I am sure we will find solutions to these problems in time. In fact, with some of my astronomer colleagues it is also pretty normal for them to spend an hour each morning skimming through 10 or so new papers posted to the arXiv.