I'm a Ph.D. student working in theoretical high-energy physics. In this field we don't rely on peer-reviewed journals. Instead, when a researchers wants to "publish" a paper she uploads it to the arXiv, and the paper appears on the site within a day or two. The arXiv is open in the sense that almost anyone can publish there [1]. Researchers in the field catch up on new research by scanning the arXiv daily for interesting papers. No one I know reads peer-reviewed journals. I know that many papers are also published in journals, but I believe this is a formality that has more to do with obtaining grants and such than with actual communication within the community. As far as I know there's no reason to publish in a journal before you become a professor.

The result is similar to the GitHub situation in many ways. Because there are no barriers to publishing, everyone makes up their own mind about which papers are interesting. If your work is relevant, others will build on it and cite you. They will discuss it in their group meeting, and so on. A scientist's reputation is then directly related to the quality of their work, as judged by the community, with no artificial barriers. This means that a self-respecting scientist would not publish a sub-par paper even though it's technicality possible to do so, because that would hurt her reputation.

So it seems to me that the situation in high-energy physics is close to ideal, with respect to ease of publishing and the social aspect of reputation. Having said that, there are certainly aspects of GitHub that I would love to see adopted.

For instance, when several researchers are writing a paper, generally no version control system is employed. Instead, at any point in time the draft is "locked" by one of the collaborators, and only that person can change it. Beyond the obvious inefficiency of this method, note that it is also difficult to track what changes were made in each lock cycle. I use diff for this purpose, but in my experience many scientists in the field aren't aware of such tools. So something that could really help is a simple way to collaborate on papers, just a basic source control system. Also, it must be possible to work on the paper in private within the collaboration, and only publish the end result.

[1] The few barriers that exist are in place to keep out the crackpots, who reduce the signal-to-noise ratio and in that sense resemble spammers.