Hacker News Reader

Comment by nightfader | Hacker News Reader

How is it not bad design? Let's say you are working in a team. Would you really want your colleagues spending a significant amount of time cloning your artifacts? Your comment is also not consistent with forgetting that one is not a developer. Even if it's my grandma, she's not gonna want to wait for 1hour to download a giant file from VC assuming she knows what a VC is. Large blobs can go into versioned object storage like GCS or S3 etc

View on HN

In Subversion at least, you'd do a partial checkout. If you don't need a particular directory you just don't check it out. If you lay out your repo structure well there's no problem. It was incredibly convenient.

I've tried many different SCM over the years and I was happy when git took root, but its poor handling of large files was problematic from the beginning. Git being bad at large files turned into this best practice of not storing large files in git, which was shortened to "don't store large files in SCM." I think that's a huge source of our availability and/or supply chain headache.

I have projects from 20 years ago that I can build because all of the dependencies (minus the compiler -- I'm counting on it being backwards compatible) are stored right in the source code. Meanwhile, I can't do that with Ruby projects from several years ago because gems have been removed. I've seen deployments come to a halt because no startup runs its own package server mirror and those servers go offline or a package may get deleted mid-deploy. The infamous leftpad incident broke a good chunk of the web and that wouldn't have happened if that package was fetched once and then added to an appropriate SCM. Every time we fetch the same package repeatedly from a package server we're counting on it having not changed because no one does any sort of verification any longer.

SCC systems that handle big files don't suffer from the "you have to clone all the history and the entire repo all the time" problem that git suffers from. At least Perfoce doesn't...

git has its place but it's really broken the world for how to think about SCC. There are other ways to approach it that aren't the ways git approaches it.

When you make a video game you want version control for your graphics assets, audio, compiled binaries of various libraries, etc. You might even want to check in compiler binaries and other things you need to get a reproducible build. Being able to chuck everything in source control is actually good. And being able to partially check out repositories is also good. There is no good technical reason why you shouldn't be able to put a TB of data under version control, and there are many reasons why having that option is great.

The versioned object storage solves nothing. If your colleagues need the files, they're going to have to get them, and it's going to be no quicker getting them from somewhere else. Putting them outside the VCS won't help. (For generated files, you may have options, and the tradeoffs of putting them in the VCS could be not worth it. But for hand-edited files, you're stuck.)

If the files are particularly large, they can be excluded from the clone, depending on discipline and/or department. There are various options here. Most projects I've worked on recently have per-discipline streams, but in the past a custom workspace mapping was common.

> Would you really want your colleagues spending a significant amount of time cloning your artifacts?

Not just the artifacts, but their entire history. That is a problem that Git has out of the box, but there is no reason it needs to work that way by default. LFS should be a first class citizen of a VCS, not an afterthought.

So how would you version a game that needs assets? These files must be versioned but can be very big, for example long cutscene videos.

Some projects need the ability to version big files, there is a good reason why perforce exists and is widely used in the gaming industry.

I am not saying that it is a better UX, but hashed/versioned blobs on S3 would mostly work depending on tooling integration.

Isn’t that essentially what git lfs is?

I believe so, but with different UX. In almost every case I expect git lfs to be better, but I can see reasons to use more custom flows.

That's building a custom version control on top of the version control you're already using.

not really, it is like building a custom storage layer for your VCS.

you are still relying only on git as a source of truth for which artefacts belong to which version.

Reply on news.ycombinator.com