Hacker News Reader

Comment by vtbassmatt | Hacker News Reader

A handful of us from the product team are around for a few hours to discuss if you're interested.

Does the virtualization work equally well for lots of history as it does for large working copies?

I have a 100k commit svn repo I have been trying to migrate but the result is just too large. Partly this is due to tons of revisions of binary files that must be in the repo.

Does the virtualization also help provide a shallow set of recent commits locally but keep all history at the server (which is hundreds of gigs that is rarely used)?

GVFS helps in both dimensions, working copy and lots of history. For the lots of history case, the win is simply not downloading all the old content.

A GVFS clone will contain all of the commits and all of the trees but none of the blobs. This lets you operate on history as normal, so long as you don't need the file content. As soon as you touch file content, GVFS will download those blobs on demand.

Thanks - that sounds perfect for lots of binary history as you never view history on the binaries, only the source files.

This is amazing, congrats. I worked on Windows briefly in 2005 (the same year git was released!) and was surprised at how well Source Depot worked, especially given the sheer size of the codebase and the other SCM tools at the time.

Is there anything people particularly miss about Source Depot? Something SD was good at, but git is not?

I just got a request today for an API equivalent to `sd files`, which is not something Git is natively great at without a local copy of the repo.

Another interesting complaint is one that we hear from a lot of people who move from CVCS to DVCS: there are too many steps to perform each action. For example, "why do I have to do so many steps to update my topic branch". While we find that people get better with these things over time, I do think it would be interesting to build a suite of wrapper commands that roll a bunch of these actions up.

How do you prevent data exfiltration? I mean, in theory you could restrict the visibility of repos to the user based on team membership/roles and so prevent a single person from unauditably exfiltrating the whole Windows source code tree. In contrast with a monorepo there likely won't be any alerts triggered if someone does do a full git clone, except for someone saturating his switch port...

What would someone do with the source code for Windows? No one in open source would want to touch it. No large company would want to touch it. Grey/black hats are probably happier with their decompilers. Surely it would be easier to pirate than build (assuming their build system scales with most build systems I've observed in the wild). No small company would want to touch it.

Anyway MS share source with various third parties (governments at least and I believe large customs in general) so any of these are a potential leak source.

This is all correct. Also, we'd notice someone grabbing the whole 300GB whether it's in 40 SD depots or a single Git repo.

Mind if I ask how you'd notice?

The article mentions relying on a windows filesystem driver. Two questions about that:

1) Why include it in default windows? It seems that 99.99% of users would never even know it existed, let alone use it

2) Does that mean GVFS isn't useable on *nix systems? Any plans to make it useable, if so?

It's not included in Windows, which is why they have a signed drop of the driver for you to install. Even internally we have to install the driver.

E- oops, missed that line. Cool, wonder if it will show up in more than Git

Yes, I was referring to the plan to include it in future Windows builds

Probably as an optional feature?

1) The file system driver is called GvFlt. If it does get included in Windows by default, it'll be to make it easier for products like GVFS, but GvFlt on its own is not usable by end users directly.

2) GVFS is currently only available on Windows, but we are very interested in porting to other platforms.

What are your thoughts on implementing something more general like linux's FUSE[1] instead? A general Virtual Filesystem driver in Windows could be used for a wide range of things and means you don't just have a single-purpose driver sitting around.

[1] https://en.m.wikipedia.org/wiki/Filesystem_in_Userspace

A general FUSE-like API would be very useful to have, but unfortunately it can't meet our performance requirements.

The first internal version of GVFS was actually based on a 3rd party driver that looks a lot like FUSE. But because it is general purpose, and requires all IO to context switch from the kernel to user mode, we just couldn't make it fast enough.

Remember that our file system isn't going to be used just for git operations, but also to run builds. Once a file has been downloaded, we need to make sure that it can be read as fast as any local file would be, or dev productivity would tank.

With GvFlt, we're able to virtualize directory enumeration and first-time file reads, but after that get completely out of the way because the file becomes a normal NTFS file from that point on.

For use cases that fit that limited virtualization, is GvFlt something to consider? Or is that a bad idea?

I'm curious what the cross-over between GvFlt may be and the return of virtual files for OneDrive in the Fall Creators Update? Is the work being coordinated between the efforts?

From your description it sounds like there could be usefulness in coordinating such efforts.

Why do you name it "GVFS" instead of something more descriptive like "GitVFS"?

This was discussed a bit in the comments in Brian Harry's last post on GVFS: https://blogs.msdn.microsoft.com/bharry/2017/02/03/scaling-g...

We're building a VFS (Virtual File System) for Git (G) so GVFS was a very natural name and it just kind of stuck once we came up with it.

Are you aware that the name was already taken[0] for something which also has to do with file systems?

[0] https://wiki.gnome.org/Projects/gvfs

Sure, a couple of questions:

1. How do you measure "largeness" of a git repo?

2. How are you confident that you have the largest?

3. How much technical debt does that translate to?

1. Saeed is writing a really nice series of articles starting here: https://www.visualstudio.com/learn/git-at-scale/ In the first one, he lays out how we think about small/medium/large repos. Summary: size at tip, size of history, file count at tip, number of refs, and number of developers.

2. Fairly confident, at least as far as usable repos go. Given how unusable the Windows repo is without GVFS and the other things we've built, it seems pretty unlikely anyone's out there using a bigger one. If you know of something bigger, we'd love to hear about it and learn how they solved the same problems!

3. Windows is a 30 year old codebase. There's a lot of stuff in there supporting a lot of scenarios.

Is it possible to checkout (if not build) something like Windows 3.11 or NT 4?

As far as I can recall, this is not possible using Windows source control, as its history only goes back to the lifecycle of Windows XP (when the source control tool prior to GVFS was adopted).

Microsoft does have an internal Source Code Archive, which does the moral equivalent of storing source code and binary artifacts for released software in a underground bunker. I used to have a bit of fun searching the NT 3.5 sources as taken from the Source Code Archive...

I recently heard a story that someone tried to push a 1TB repo to our university Gitlab which then ran out of disk space. Sure, that might have been not be a usable repo but only an experiment. Still, I would bet against the claim that 300GB is the largest one.

1 TB of code?

I'd sure like to run that as my operating system, browser, virtual assistant, car automation system, and overall do-everything-for-me system...

300 GB is not the size of _the repository_. It's the size of the code base - the checked out tree of source, tests, build tools, etc - without history.

It's certainly possible that somebody created a 1 TB source tree in Git, but what we've never heard of is somebody actually _using_ such a source tree, with 4000 or more developers, for their daily work in producing a product.

I say this with some certainty because if somebody had succeeded, they would have needed to make similar changes to Git to be successful, though of course they could have kept such changes secret.

I'm currently investigating using GitLFS for a large repo that has many binary and other large artifacts.

I'm curious, did you experiment with LFS for prior to building GitVFS?

Also, I know that there is an (somewhat) active effort to port GitVFS to Linux, do you know if any of the Git vendors (GitLab and/or GitHub) are planning to support GitVFS in their enterprise products?

Yes we did evaluate LFS. The thing about LFS is that while it does help reduce the clone size, it doesn't reduce the number of files in the repo at all. The biggest bottleneck when working with a repo of this size is that so many of your local git operations are linear on the number of files. One of the main values of GVFS is that it allows Git to only consider the files you're actually working with, not all 3M+ files in the repo.

> One of the main values of GVFS is that it allows Git to only consider the files you're actually working with, not all 3M+ files in the repo.

That is an excellent point. Thanks!

We at GitLab are looking at GitVFS but have not made a decision yet https://gitlab.com/gitlab-org/gitlab-ce/issues/27895

Very cool blog! As I understand, you dynamically fetch a file from the remote git server once for the first time I open the file. Do you do any sort of pre-fetching of files? For example, if a file has an import and uses a few symbols from that file, do you also fetch the imported file beforehand or just fetch it when you access it first time?

We don't currently do that sort of predictive prefetching, but it's a feature we've thought a lot about. For now, users can explicitly call "gvfs prefetch" if they want to, or just allow files to be downloaded on demand.

For now, we're not that smart and simply fetch what's opened by the filesystem. With the cache servers in place, it's plenty fast. We do also have an optional prefetch to grab all the contents (at tip) for a folder or set of folders.

What's the PR review UI built in?

Custom JQuery-based framework, transitioning to React.

Actually, I think we finished the conversion to React :). So, React.

Taylor is the dev manager for that area so I'm inclined to believe his correction :)

What was the impetus for switching to git?

More or less:

- Availability of tools

- Familiarity of developers (both current and potential)

Any plans to port GVFS to Linux or macOS?

Why not TFS?

https://news.ycombinator.com/item?id=14411588

"A handful of us from the product team are around for a few hours to discuss if you're interested."

Thanks!

This is a little off-topic, but why can't Windows 10 users conclusively disable all telemetry?

(I consider the question only a little off-topic, because I have the impression that this story is part of an ongoing Microsoft charm-offensive.)

Haha, no answer, as expected. HN got butthurt as well lol.

I knew there was a risk of getting downvoted, but I was surprised that it went to "-4".

Reply on news.ycombinator.com