Hacker News Reader

Comment by VladRussian2 | Hacker News Reader

>With perforce, you can reasonably expect to run into this brick wall somewhere in the neighborhood of terabytes of metadata and dozens of transactions per second. That changes depending on what sort of beastly hardware you are willing to throw at your version control team.

>Git of course hits a brick wall much sooner, somewhere around single-digit gigabytes of data (depending heavily on the average size of every object in the DAG), even ignoring throughput.

if you use Git the same way you'd use Perforce or Mercurial - one central server handling everything ... you'd better stay with Perforce or Mercurial. Corporate dev process is usually built around notion of "central" repo (and they equate it with "master") - just replacing Perforce server with Git server wouldn't change anything.

View on HN

That is certain. If you are trying to run at a very large scale with a single repo, Perforce (or apparently Mercurial) is the way to go.

The problem is "how do you grow even further?" There isn't a perfect solution to that (yet), but I think the only existing workable (although absolutely imperfect) solution is splitting your codebase into many small repos.

Thankfully very few companies need to worry about this problem right now. A single Mercurial repo is apparently scaling for Facebook currently, and few companies will manage to stress Perforce.

> If you are trying to run at a very large scale with a single repo, Perforce (or apparently Mercurial) is the way to go.

that is the problem - single logical repo in Perforce/Mercurial does mean single physical repo. Which obviously causes issues at very large scale. Even at normal enterprise scale :)

Git solves it through kind of distributed scaling where many operations can be performed on local physical repos without ever hitting the central. With centralized solutions, like Perforce, Mercurial, etc... most of the operations are performed on central server and your ability to scale vertically hits the ceiling pretty soon.

Compare the simplest case - each dev updating his local workspace/repo would hit the central server in Perforce where is with Git you can (and normally would have) a small set of downstream repos which the updates would be propagated/distributed through. You can branch left and right in Git in your local and team's repos without "master" repo and "central" server involved (while all your branch activity in Perforce is happening right on the central server in the central repo). The same happens for synchronizing your work inside the team - no need to hit the central. Etc...

Mercurial is a DVCS, just like git. It has no concept of a central server.

>Mercurial is a DVCS

my bad. 5+ years since i worked with Mercurial. Digging deep into painful memories, my Mercurial PTSD from that time is absence of in-repo branching - need to clone which is a killer for very large repo we had and no partial commits - again a killer aggravated by the above mentioned absence of in-repo branches. Both issues made working with large repo unreasonably and unnecessary hard.

hg certainly has had branches since 2008 at least.

What do you mean by partial commit? I can guess but I would rather know. :-)

I am sure Mercurial has always had (named) in-repo branches, so you were using Mercurial in strange manner.

There was a time when the developers advised using parallel clones rather than named branches, though. I never really understood why.

These days, the standard practice is probably to use bookmarks, which are more or less like Git branches.

These days, the standard practice is probably to use bookmarks, which are more or less like Git branches.

No. You have the choice to use either named branches or bookmarks, and which one you choose is a matter of your workflow. Note that even a Git-like workflow does not necessarily require the use of bookmarks.

Mercurial is still designed under the assumption that people will use named branches and the Git-like aspects of bookmarks exist to ease transition from Git: http://www.selenic.com/pipermail/mercurial/2014-January/0464...

Personally, I use Fossil-like private branches [1] in order to facilitate local work that I may still want to reorganize before pushing it to a shared server; I use bookmarks solely for local tags.

[1] Emulated by putting a named branch in the secret phase. I actually have a simple extension to automatically make all branches that start with a dot (e.g, ".tempstuff") secret.

> With centralized solutions, like Perforce, Mercurial, etc...

I think you're incorrectly putting mercurial and perforce in the same bucket. Mercurial is a lot more like Git than perforce in that it's a DVCS. Facebook made it scale by using a lot of centralized caching, but most operations (diff, looking through history, branching, etc) can be done locally.

Git doesn't solve the issue, it suffers from it, that's the point. And as others pointed out already, mercurial is a DVCS. It just isn't as shitty as git, that's all.

Mercurial itself runs into issues way before git does, it's just that Facebook has essentially gutted a bunch of things out Mercurial in order to make the resulting thing fast (punting on things like computing status, downloading diffs, etc).

No, git runs into issues before mercurial does. Just saying the opposite of reality doesn't make it so.

Did you even read the post? Stock git is significantly faster than stock mercurial, it's only after they hack it up that they make it faster.

There are some operations in which git is way slower than Mercurial. Two examples are cloning and blame/annotate. git's data structures are optimised for some operations, but not for others.

Did you? It isn't about speed, it is about scaling.

At what point do we say "enough" though? I mean, start from the absolute maximum: how would one make a system to control all of the source code in the world?

For all of the source code in the world that is source-controlled, it exists in separate repos. It doesn't exist in one repo, and where there are inter-project dependencies, it is the dependency consumer's responsibility to keep abreast of the changes in the dependency and integrate them as necessary. It is self-organizing.

So, if Google and Facebook have grown beyond what can be done with a single repo, either they are not being mindful of their engineering practices (which puts the public at significant risk), or they are approaching a scale that is more similar to "world scale" than it is "corporate scale".

Which I would also say is putting the public at significant risk.

This is a very valid point. It is not clear at all that a single repo for the world is desirable in the long run, despite some of the individual advantages that sort of setup has. It is part of the reason that I don't think we will see the "next generation" of VCS anytime soon. (Which in turn makes me think that "holding out" for those systems, hoping they will rescue you from your scaling problem, is a bad decision. It is best to move to multiple repos sooner rather than later).

Reply on news.ycombinator.com