Hacker News Reader

Comment by tankenmate | Hacker News Reader

"Git wasn't designed for such vast numbers of developers—more than 20,000 actively working on the codebase. Also, Git wasn't designed for a codebase that was so large, either in terms of the number of files and version history for each file, or in terms of sheer size, coming in at more than 300GB. When using standard Git, working with the source repository was unacceptably slow. Common operations (such as checking which files have been modified) would take multiple minutes."

These sentences in some parts gloss over the details, and in others it is flat out wrong. Git was designed for tens of thousands of developers (the Linux kernel), it was designed for huge numbers of files (but large numbers of files works fine on Linux due to the dentry cache, it sucks on Windows because they don't have a cache that has the same behaviour as the Linux dentry cache). Admittedly it is slow for files that are large in size, but it was designed for source code; what sane developer would have source files that are hundreds of MB in size?

That said, monorepos with several dozen software projects can be very slow due to O(n) functions such as git blame and the like. It's true that git wasn't designed for large numbers of projects in one repo that are only indirectly shared by library code and the like.

Unfortunately that was not written in the article.

View on HN

> in others it is flat out wrong. Git was designed for tens of thousands of developers (the Linux kernel),

This is easy to check, and the article is right. The Linux kernel does not have tens of thousands of active developers. Over a little more than the last year:

    titan:~/src/linux geofft$ git log v4.8..v4.14 --format='%aE' | sort -u | wc -l
    4857

If you look at just the most recent release cycle:

    titan:~/src/linux geofft$ git log v4.13..v4.14 --format='%aE' | sort -u | wc -l
    1839

Alternatively, if you look over the last ~year, but at people who have more than five commits (chosen arbitrarily):

    titan:~/src/linux geofft$ git log v4.8..v4.14 --format='%aE' | sort | uniq -c | awk '$1 > 5' | wc -l
    1814

A codebase with 20,000 actively working on the code base is at least an order of magnitude more than the Linux kernel. If they're committing patches every single day, probably more than that. I have, last I checked, four commits in Linux: two for a summer internship in 2009 and two doc fixes. That certainly contributes to the scale of Linux the kernel in the sense of "there are so many people working on it!", but not in the sense of being an "active developer" of the git repo. I was fine sending in my patches to a mailing list and hearing back at some arbitrarily later point, not triggering any CI, not asking anyone else to collaborate on my work, etc. That doesn't work if you have 20,000 engineers on the same codebase writing code every day.

> Admittedly it is slow for files that are large in size, but it was designed for source code; what sane developer would have source files that are hundreds of MB in size?

Sometimes that's the best way to get things done? The best Debian workflow I've ever worked with (and I've worked with a lot) involved actually committing binary .debs to SVN alongside their source code, because that meant that the SVN revision number was the single source of truth. There wasn't some external artifact system, nor was there a risk of picking up the wrong packages from an apt repo when rebuilding an older SVN revision.

I'm not defending this as pretty. But I will defend this as sane. It got the job done reliably and let me work on actually shipping the product and not yak-shaving workflows.

+1 for the research and the distinction between sanity and purity.

Good stuff. Came into this thread expecting the top comment to be a know-it-all answer trying to "debunk" the article, as so commonly seen on HN. This is a perfect, well-researched response.

https://news.ycombinator.com/item?id=4693920

Interesting read

> actually committing binary .debs to SVN alongside their source code, because that meant that the SVN revision number was the single source of truth

This is totally armchair-devops, but it sounds like a more ideal case would be CI that runs based on your commit. In my experience, developers get lazy and are likely to commit binaries that dont match the code or miss some dependencies or steps.

I completely agree with your point, using the tools as intended isn't always the best way to get things done. Video games are an easy example, code is often dependent and intermixed with binary assets and you want those to be versioned together.

If you actually commit binaries to help with the CI, then those binaries should be installed by the CI. I'm not saying it's a good idea to do it this way, but if you really want to for some reason, then that's the way to enforce no steps or binaries are missing. (And if devs forget to add something, the build would just fail)

Apologies if I wasn't clear, that's what I was getting at. Commit the code, CI generates the binaries. I guess you could commit the binaries if you wanted, but a separate caching repo would make things more manageable. You'd still be using the commit to reference the binary. But this is ideal, and there are good, practical reasons doing what he describes.

For games the binary data isn't from code. They're textures and models generated in a package (like Photoshop). The bits of code are often tightly coupled with the textures--like a procedural waterfall that uses still images. You'd need to version both of those together.

I mean, it's certainly true that at least once I found that another developer had updated the .debs and forgotten to update the sources, and I had to attempt to reconstruct what he did because I needed to add another patch on top. So you're absolutely right that some better CI would have helped here.

But for the product we were building, it was important that developers were able to test their .debs before committing to the repo. That means that CI can't build post-commit: you have to have some way to take a local modification, turn it into a .deb, test it and see if it does what you want, and then commit that.

I've thrown something together for my current job that supports both building a package interactively for testing and building one in the CI pipeline post-commit (using a git repo as the source in both cases) but it's definitely more cumbersome, and I haven't figured out quite how I want to fix it.

(Also, yes, games are a more obvious example of this, but there are lots of purists who've never worked in game development who are somehow convinced that assets aren't code and shouldn't be in the code repo. Having not worked in game development myself either, I figured it was safer not to invite that argument.)

Are private branches that the CI can pick up also not a workable solution for your usecase?

That's what I'm doing now at work, and it's sort of silly to git push, wait for CI, notice that it failed, fix, git push, wait for CI, .... in an SVN world, that would have meant one commit in the monorepo (since there's no such thing as temporary branches, and revisions are global across all branches) for every typo I make during development. You could make it work with git, certainly, but on balance the check-in-a-.deb approach worked fine and didn't require spending more time writing tooling.

You could try gerrit + e.g. Jenkins. When pushing to review the CI can pick up the commit, build it and optionally run tests and give +1/-1 verified.

The commit is merged only after a developer will give +2 for the review.

I think this is all missing the point. Do they want to store, say, the entirety of the base Windows OS in a single git repo? If that's the case, sure, git isn't a great fit for their use case, and they should either use something else, or find a way (as they're doing?) to change git to fit their needs.

I can't see MS having tens of thousands of developers working on any single component of Windows, so I'm guessing they _do_ want a giant monorepo. If they were to break it up into separate repos per OS component, I doubt they'd have scaling issues (of course, breaking things up introduces coordination and dependency issues).

I can't see it being easy to take a 20 year massive code base and split it up into components that are so isolated enough they can live in different repositories smoothly.

As they say in: https://blogs.msdn.microsoft.com/bharry/2017/02/03/scaling-g...

The first big debate was – how many repos do you have – one for the whole company at one extreme or one for each small component? A big spectrum. Git is proven to work extremely well for a very large number of modest repos so we spent a bunch of time exploring what it would take to factor our large codebases into lots of tenable repos. Hmm. Ever worked in a huge code base for 20 years? Ever tried to go back afterwards and decompose it into small repos? You can guess what we discovered. The code is very hard to decompose. The cost would be very high. The risk from that level of churn would be enormous. And, we really do have scenarios where a single engineer needs to make sweeping changes across a very large swath of code. Trying to coordinate that across hundreds of repos would be very problematic.

> I can't see it being easy to take a 20 year massive code base and split it up into components that are so isolated enough they can live in different repositories smoothly.

I can't see it being desirable either, they already have to deal with outdated external API, surely they don't want outdated internal API. A single repository lets you replace an API throughout the codebase at once and remove the old one entirely, if you start "modularising" then all "internal API" are suddenly external with all that entails.

PM for Git at Microsoft here. We explored splitting it up. It's a 300GB repository and it's been a monorepo for the last 20 years. Splitting it up logically would take a lot of time (to put it mildly) and development would stall while we did it. And once we did, we would have 100 3.5GB repositories? 3500 100MB repositories? Neither of these are particularly appealing and getting changes checked in atomically across multiple repositories is insanely challenging. There's no doubt that we would need to build tooling to make this work for us. (We did actually explore this direction, but ultimately decided that it would be too much work for too poor an experience.)

Instead, we decided to - as you put it - change Git to fit our needs.

What were you using before this?

We were using a mix of tools throughout Microsoft: several teams were using Source Depot, which is an internally developed centralized version control system. It was built to handle large teams, like Windows and Office. It was the precursor to Team Foundation Version Control, which is the centralized version control system available in TFS and VSTS, which is also capable of scaling to large projects, and many organizations within the company were (and are) using that.

GVFS is part of our effort within the company to standardize on a single set of best-of-breed tools, and use the same tools inside Microsoft that we deliver to customers. So we're adopting one engineering system throughout the company and we're moving everybody to Visual Studio Team Services.

It's my understanding that Source Depot was not internally developed but rather was a fork off Perforce which MS purchased a source license to some years back.

True. Source Depot originated as Perforce and is a fork; it had (at one time) a pretty large team of developers working on it and it has been heavily, heavily modified in the many years since.

Changing Git to fit your needs probably just makes this monorepo function on borrowed time. Fixing the symptoms with band-aids will work for now, but at some point, something will probably have to be done about this 300GB god-repo. Any insight in to what/when/if something is going to be done about that?

I'm not the right person to ask about Windows organization, I'm afraid. But I will say that the nice thing about moving to Git is that its lightweight branching makes refactoring much easier.

Does Git on MSWindows use a dentry like cache in user land to speed up filename lookups? Does it use a stat() equiv cache? Is there a reason that such caches weren't put into ntoskrnl.exe? Would you be able to give us a brief list (say top 5) of changes that sped up git on windows? And which ones had the most effect on super sized monoreops. Thanks!

I have seen poor performance of git on Linux in very large repos, so I'm not super convinced that the dentry cache magically makes things better.

In particular, if my repo is big enough, I often don't have the entire tree in memory (because I'm doing other useful things with memory and caches got evicted). core.untrackedCache makes things a little better, but it's still not great.

Question: what's the `sort -u` for? You're piping to `wc -l`.

"u" stands for unique, so after sorting it skips identical rows

Oh! TIL, thanks!!

Dude teach me how to pipe liKe you

>These sentences in some parts gloss over the details, and in others it is flat out wrong. Git was designed for tens of thousands of developers (the Linux kernel),

Are there really ten thousand active developers on the Linux Kernel?

According to [1] there were ~ 2,000 over a 15 month period. It's an order of magnitude difference that seems like it would have an impact.

[1] https://arstechnica.com/information-technology/2015/02/linux...

And even then, the kernel doesn’t follow anything like the approach to distributed development that your average Github project does. The kernel consists of a bunch of maintainers who each have their own intake branches, to which other people submit PRs or patches as that maintainer wills; and then the mainline kernel, which only takes contributions via patches from one of the maintainers.

The nearest thing you could compare the development model to is the set of base packages for the BSDs: each subsystem/package has its own upstream development (like the Linux subsystems with their maintainers); and then Linus serves as the release engineer, taking updates to those packages and merging them into the release. (I compare to the BSDs specifically because Linux kernel subsystems, like non-cross-platform BSD base-system packages, have no other life outside of their potential integration into the release. People aren’t using the upstreams directly, other than in development of the new feature.)

I mean, that's the model git was designed for, and it's a model that was designed for a large number of people (by using gatekeepers doing bulk updates to mitigate the flood of small ones).

So I don't think raising the way linux uses git by its original intended users really contradicts the idea that it was designed for a large number of developers?

The article above was actually wrong and has been updated to show that Microsoft has 3,000 active developers instead of the original 20,000 that is quoted here. So it's only a difference of 50%, not a full order of magnitude.

Although you should consider how active those devs are... I'm imagining its far more active than Linux kernel devs... so the activity levels can still be an order of magnitude higher

It's probably more active than some Linux kernel developers (those who don't work on the kernel exclusively), but the majority of Linux kernel developers are employed to do kernel work and so I don't see why there would be a difference in amount of code output by either sets of developers.

I develop a desktop app with lots of test documents (test inputs, outputs) that are tied to specific revisions in order to work with the test suite that uses them. None are huge but there are many and some are frequently changed (2GB working copy, 15gb repo, 100k commits). They could be put in some (very) complex hierarchy on a network share, but having them in vcs is so convenient. If rev 9999 changes the code so that the expected output testfileA differs then you update the expected output in the same commit. If you don’t have the file and you want to be able to clone+build+test any revision then you still need the results for 9998 also on the share.

This scenario is nicely handled in svn, mercurial, tfvc and similar. If this lets git do it then it looks like a great idea.

Game developers with lots of churn in huge binary assets also have the same problem, but probably even worse.

Those kinds of use cases are what Git LFS [1] was created to help with. The major Git vendors support it (Bitbucket, GitHub, GitLab). I'm not as certain about the open source implementations.

[1] https://git-lfs.github.com/

I dabbled a bit with lfs but it seems specifically geared towards a few very large resources that you need to manually take care of syncing etc.

What we need to move off large repos in tfvc or svn is something where the inconvenience is small enough that it doesn’t weigh heavier than the benefits of git. LFS was not that, this could be.

Git LFS doesn't solve the gamedev/large asset problem.

The issue is that these giant binary files cannot be merged. Because of this you need a locking mechanism(which SVN and Perforce both support).

Without that you have two people working on the same file and someone having to throw away work which makes for a very unhappy team.

DVCS is great but not every problem domain maps to it.

git-lfs supports locking, beginning in 2.0.

Man, had to really dig to find that. Ctrl-F "lock" didn't hit anything on the landing page or main docs page.

Even then it looks like an implementation that misses the mark. Both Perforce and SVN mark "lockable" files as Read-Only on a sync of the local copy. It looks like all this does is prevent a push of the file which will be the same result as if someone merged it ahead of you.

Without that guard in place it makes it really easy to make a mistake and then have changes to a binary file that will collide. Having a checkout process as part of locking workflow is pretty much mandatory.

[edit]

Nevermind, I finally found this[1] that explains Git LFS does it correctly, kudos! Could do with some more detailed documentation since it looks partially implemented from the man pages.

[1] https://github.com/git-lfs/git-lfs/wiki/File-Locking

This is why we’re mostly stuck in Perforce hell in the games industry.

Git LFS' Achilles' heel is pruning old data.

I made a stupid decision to use git-lfs to track a pre-initialized Docker database image (using LFS to store initial_data.sql.bz2). Now I have to live with few dozen gigabytes of the past I don't really need (it's a small database).

I know I can rewrite the history and then find something to garbage collect unreferenced blobs (haven't investigated it). The repo allows doing so as it only keeps those dumps, the Dockerfile and a pair of small shell scripts. And, most importantly, it has only 2 users - me and my CI. If it would be large assets and lots of code, checked out and worked on by a large team - I guess, rewriting the history isn't an option.

Storage is cheap, but still...

Game devs just use Perforce (for binaries, or everything). Sometimes Git in combination.

The Windows repo is massive, at least 100GB. Consider that the component equivalent to Linux, the NT kernel, is but a small part of the whole thing. And it is indeed a single large project, where every commit has to leave the entirety of Windows in a good state. So if you change some widely used function, you need to fix every single caller in all of Windows. Also, the build system evolves along with the product, so it is checked into the repository as well. The right version of the build system is whatever is in the timestamp you are building. So there’s your seemingly elusive good reason for keeping large binaries in the repo. And while I worked there, the object files from the nightly build were also checked in, so you could rebuild mostly-unchanged components quickly.

The alternative to doing it this way is to introduce version management between components, which is totally unnecessary complexity. Windows is one product and all of the components (or subsets thereof) ship together. As far as internal-only interfaces are concerned, version N does not need to work with N-1 or N+1, only N. That is a massively simplifying assumption that makes a lot of things much easier.

There is a reason why all the big tech companies operate this way. Microsoft, Google, and Facebook, all have one giant repository with their entire product.

> There is a reason why all the big tech companies operate this way. Microsoft, Google, and Facebook, all have one giant repository with their entire product.

Cool, but that doesn't explain why Office and Windows need to share the same repo. Ditto for gmail and chrome os. Ditto for fb messenger and whatever face recognition algorithm they use.

Office and Windows don't share the same repo...

I see how my post could have led to that interpretation. That’s not what I meant. There are certainly many many repos in use across Microsoft, for all sorts of different projects. Office doesn’t share the same repo as Windows. And neither does the toolchain (compiler/linker/etc.); only the binaries are checked into the Windows repo. Those are truly separate products, which need to work across different versions of Windows.

I worked at a game company for a while and one of the interesting things there was that there really wasn't a good way around checking in binaries to the repo. First off, a lot of your "source" assets are properly in binary form (images, for example), because that was the most compact form for them to exist, by orders of magnitude, and also because trying to generate them from some other non-binary representation would add tremendous overhead to development work. On top of that, building the entire product from "source" might involve tens to hundreds of thousands of hours of CPU/GPU time. So you just didn't do that. Instead you built everything in parallel and checked the results in somewhere, that would then get bundled together into the final product. There are ways around doing things this way, for sure, but given that you can just go pay a fairly insignificant amount of dollars to some company and get a VCS that won't choke.

IMHO it is a major failure that git doesn’t scale for org-wide mono repos. Software development process should not be dictated by limitations in the VCS. It surprises me that others are so willing to settle.

Would you be willing to elaborate on the pros/cons of mono repos vs what git encourages? It seems to just be the wrong fit for a given use case, but works very well others.

That's a good and valid point, which deserves a conversation undoubtedly - however, there are a lot of businesses out there who have a large monolith, legacy application gigabytes in size where the cost of slicing and dicing into smaller repositories has to be driven by requirements and not technology.

My current client has a 17-year-old multi Gb TFS monolith. New components go into git and as features evolve it's common for the respective team to hive them off into they're own git repos as stand-alone components. Nevertheless TFS is painful and if I could have an effective git repo with the monolith too, then life would be far easier.

Is there a way to split a TFS repository, surely you no longer need the past 10 years of commit history on a daily basis - archive it or something?

Also how feasible is migrating from TFS to Git?

Very, thanks to git-tfs [1]. My company is in the midst of a move from on premise TFS/TFVC to cloud based VSTS/git. We've run into some projects that managed to somehow mangle their branch history so to the point that we gave up and only migrated the current checkin to git, but most projects pull out the full history and all branches with no trouble.

The only real complaint I have is speed. Dunno if it's our TFS server or the nature of TFS, but every checkin pulled into git takes anywhere from 5-30 seconds. In large projects with long histories you can easily spend 30+ minutes exporting history before running into a problem that prevents the export from completing.

[1]: https://github.com/git-tfs/git-tfs

> In large projects with long histories you can easily spend 30+ minutes exporting history before running into a problem that prevents the export from completing.

If it makes you feel any better I had to spend two continuous weeks exporting our huge TFS monolith before I managed to have a working git copy.

But yeah, Git-TFS is a godsend.

Is there a VCS that does?

When I was at Amazon, Perforce couldn't scale (it had already been partitioned into a few depots, but those had reached their reasonable capacity). Amazon moved to a per "module" git-repo system. It was convenient to push across multiple repos simultaneously, but it wasn't a deal breaker when we lost it (I never heard of any workflows being completely unworkable following the transition).

Google has one internally, and Facebook has done a ton of work on Mercurial to build something similar, doing things like replacing files on disk with memcache servers running close to the dev machines. But I don't think there's anything that works "out of the box" at that scale.

For quite a while at Google there have been git extensions/tools for working with Piper. They're pretty widely used, but haven't been "officially" supported.

There's a new project that allows you to use hg with Piper/CitC; I've been using it for about a month and it's been working really well. No clue how it works, though.

Piper: https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...

Unfortunately, it's not available externally.

Even working at Google, my jaw still dropped at this:

    Google's codebase is shared by more than 25,000 Google software
    developers from dozens of offices in countries around the world.
    On a typical workday, they commit 16,000 changes to the codebase,
    and another 24,000 changes are committed by automated systems.
    Each day the repository serves billions of file read requests,
    with approximately 800,000 queries per second during peak traffic
    and an average of approximately 500,000 queries per second each
    workday. Most of this traffic originates from Google's
    distributed build-and-test systems.

Mostly I felt terribly unproductive — my changelist generation rate is way below average.

which means nothing, e.g. by loc your changestats are order of magnitude higher than mine

Wow! Thanks for sharing. I hadn't contemplated how Google managed their source code. Sounds like Piper has some nice features!

Lots more details here [1] if you're interested.

[1] https://www.youtube.com/watch?v=W71BTkUbdqE

The number of engineers and commits is probably in the same ballpark at Amazon. With a DVCS, far less operations are done server side, so I would imagine the read requests are an order of magnitude or two lower. They have tooling to get a global view across all repos, however there are no cross-repo atomic operations (those can be managed in their build system in a relatively robust way).

What advantage does a company wide mono repo provide?

The ability to have something like CitC (or this post's git virtual filesystem) is certainly one big advantage -- no need to clone new packages, they're right there in your "local" source tree. Bazel (blaze) is another, particularly when coupled with working at HEAD.

My experience with farms of git repos is that the lack of atomic operations over many tiny repos leads to things like version sets and having to periodically merge dependencies. I've worked on teams where that was inevitably neglected during hectic periods resulting in painful merges of large numbers of changes. That problem simply doesn't exist with working at HEAD and high quality presubmit test automation/admission control. The single repo also allows for single code reviews spanning multiple packages which makes it MUCH simpler to re-arrange code (Bazel again helps here since a "package" is any directory with a BUILD file). Package creation is lighter weight for the same reason, and has fewer consequences for poor name choices since rearrangement is easy and well supported by automated tools.

Sharing one build system where a build command implicitly spans many packages also results in efficient caching of build artifacts and massively distributed builds (think a distributable and cacheable build action per executable command rather than a brazil-build per package). Each unit test result can be cached and only dependent tests re-run as you tweak an in-progress change. This is fantastic for a local workflow (flaky tests can be tackled with --runs_per_test=1000, which with a distributed build system is often only marginally slower than a single test run). Also, you can query all affected tests for a given change with a single "local" bazel query command. The list goes on from here -- I keep thinking of new things to add (finer grained dependencies, finer grained visibility controls, etc.).

It's not that you can't build most of this for distributed repos, but I'd argue it's harder and some things (like ease of code reorg) are nearly impossible.

Subjectively, having worked with both approaches at scale, Google's seems to result in much better code and repo hygiene.

Can you please quote with ">" at the start of the paragraph instead of using a code snippet? This is unreadable on mobile; the text is 3x as wide as the (scrolling) viewport.

> IMHO it is a major failure that git doesn’t scale for org-wide mono repos.

git was (initially) written by Torvalds to serve his own purpose -- he needed a new VCS for Linux. Let's say that it "doesn't scale for org-wide mono repos". Unless that was a design goal (and I'm pretty sure it wasn't), how is that a "failure"? It apparently works just fine for the kernel developers.

Yet because it didn't work well for Microsoft's 300 GB worth of tens millions of lines of code it is somehow a "failure"?

Apparently Microsoft's own TFS is a failure too, then, because it seems that it wasn't up to the task either.

There's also the popularity factor... not many care(d) about TFS. I would refrain from calling most pieces of technology a "failure", git especially.

When your screwdriver does a bad job at pounding nails, you're doing it wrong.

PM at Microsoft here. TFVC is completely different, though.

(To quickly clarify nomenclature: TFVC is "Team Foundation Version Control", the name of the centralized version control system in TFS and VSTS, which are our on-premises and cloud hosted development platforms, respectively. They both support both TFVC and Git - Git with GVFS, and they do more than just version control, hence the naming clarification. Apologies for all the three letter acronyms.)

Anyway, TFVC is totally different and a centralized version control system with all the workflows that go along with it like expensive branching. The desire to move to Git opens up all the awesome workflows that Git offers.

The impetus here was to standardize the entire company on a modern set of development tools - one engineering system across the company - hosted in Visual Studio Team Services. We could have moved them to TFVC, it's remarkably similar to its predecessor, "Source Depot" which is a Microsoft internal tool that the Windows team was using.

But that's a lateral move. There aren't really any benefits to TFVC over Source Depot except that we sell and support one to end users and don't with the other. So that has some nice organizational benefits but not enough to warrant moving everybody and the build/release farms over.

But moving to Git unlocks all sorts of new workflows - lightweight branching, pull requests, all that good stuff that everybody who uses Git is totally used to.

Anyway, that's the background on Git vs TFVC. And, no, I totally agree that Git not supporting giant monorepos is not a failing, per se. But it is a limitation, and thankfully one that we're working to help overcome.

well what would be cool if TFS could be installed on Linux and use a different database than MSSQL (PostgreSQL, MySQL or MSSQL) (I mean search uses elasticsearch anyway...). Even better if it would be more close to the online service. And way better if it came with something like gitlab-runner.

Can you clarify what you mean by "more close to the online service"? We update TFS with the bits from VSTS pretty regularly; are you looking for a faster cadence or am I misunderstanding what you're saying?

well some views are looking more clean on the online version. i.e. the project overview page.

currently I'm evaluating git hosting on-premise and another downside is, is that git tfs takes a lot of horse power (this looks unsuitable for a small team)

What I find more surprising is the length they went through to make git work for them.

It would probably have been better if they had assigned a team of highly skilled engineers to design a distributed source control from scratch specifically to solve the problems faced by the windows team.

After all, `git` itself was developed for Linux because Linus was not satisfied with any of the existing solutions.

Why would that have been better?

(Why in the world would that have been better?)

For the same reason that git turned out better than svn for a lot of use cases.

It seems like Windows is not really one of these use cases.

Tens of thousands of developers pushing changes is not the same as the Linux development process, at all. The key is in the word “actively.” Even Perforce couldn’t keep up with that type of load in a similar situation with Google’s monorepo, forcing them to develop a Perforce-compatible alternative.

It’s tough to scale VCS to very large teams, regardless of the software. No need to stand up for git.

Your comment reminds me that Microsoft should really double down on replacing NTFS. Even Apple has moved off HFS+ with their latest OS.

They unfortunately removed the ability to create ReFS disks from Windows 10 Pro and lower. Hopefully they reverse that decision soon. That said I'm not sure how stable it was considered.

I agree that these sentences gloss over the details - and rightly so: it's an article in Ars, after all, not a journal article. We (Microsoft) have published a lot[1] about our journey to put Enterprise-scale codebases, like the Windows codebase, into Git. I think you'll find that they expand upon some of the things that are glossed over.

And I would encourage you to read them, so I won't belabor the details here, but to give you just one example: git blame is not the issue that we're most concerned about. The basic, highest priority functionality in Git - things like git add - are the the issues we're most concerned about.

git add is O(n) on the number of files in the repository. When you run `add`, git reads the index, modifies the entry (or entries) that you're changing, and writes the index back out.

The Windows repository has about 3.5 million files[2]. Running `git add` - when we started thinking about moving Windows into Git - took minutes.

Can you imagine running `git add` and having it take 10 minutes?

Now obviously there are some inefficiencies here - there's quadratic operations and the like that went in assuming that nobody would ever put 3.5 million files into a Git repository. And we've been cutting those out over time[3].

Thankfully, Git does have some functionality to support very large repositories - using shallow clones, sparse checkouts and the like. We've added narrow clones, to only download portions of the tree, and automation to handle this automatically without user intervention.

That's the scaling work that we're doing with GVFS. And these changes bring the P80 time for git add down to 2.5 seconds[4]. We've been contributing these changes back to git itself, and we're thrilled to work with industry partners like GitHub who are also interested.

Sorry to go on - version control is a passion of mine, as it is for many of us working on Visual Studio Team Services. Your conclusion is very much correct: Git wasn't designed for this from the beginning. Thankfully, software isn't immutable, so we're scaling it up.

[1]: http://gvfs.io/

[2]: https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-large...

[3]: https://blogs.msdn.microsoft.com/devops/2017/05/30/optimizin...

[4]: https://blogs.msdn.microsoft.com/devops/2017/11/15/updates-t...

The article was modified to "— more than 3,000 actively working on the codebase".

Reply on news.ycombinator.com