Hacker News Reader

Comment by flohofwoe | Hacker News Reader

Even ancient svn works much better out of the box for large binary files than git (e.g. a 150 GB working directory with many big Photoshop files and other binary assets is no problem in SVN).

What does SVN do differently than git when it comes to large binary files, and why can't git use the same approach?

I also don't quite understand tbh how offloading large files to somewhere else would be fundamentally different than storing all files in one place except complicating everything? Storage is storage, how would a different storage location fix any of the current performance and robustness problems? Offloading just sounds like a solution for public git forges which don't want to deal with big files because it's too costly for them, but increased hosting cost is not the 'large binary file problem' of git.

(edit: apparently git supports proper locking(?) so I removed that section - ps: nvm it looks like the file locking feature is only in git-lfs)

View on HN

Completely different design. Git is intended to be fully distributed, so (a) every repo is supposed to have the full history of every file, and (b) locking is meaningless.

People should use the VCS that's appropriate for their project rather than insist on git everywhere.

> People should use the VCS that's appropriate for their project rather than insist on git everywhere.

A lot of people don't seem to realise this. I work in game dev and SVN or Perforce are far far better than Git for source control in this space.

In AA game dev a checkout (not the complete history, not the source art files) can easily get to 300GB of binary data. This is really pushing Subversion to it's limits.

In AAA gamedev you are looking at a full checkout of the latest assets (not the complete history, not the source art files) of at least 1TB and 2TB is becoming more and more common. The whole repo can easily come in at 100 TB. At this scale Perforce is really the only game in town (and they know this and charge through the nose for it).

In the movie industry you can multiply AAA gamedev by ~10.

Git has no hope of working at this scale as much as I'd like it to.

Perforce gets the job done but it's a major reason why build tooling is worse in games.

Github/gitlab is miles ahead of anything you can get with Perforce. People are not just pushing for git because they ux of it, they're pushing git so they can use the ecosystem.

What do you mean “build tooling is worse in games”? I’ve worked in games for 20 years and used both perforce and got extensively across small, medium, and large projects.

There’s nothing technically worse about perforce with regard to CI/CD, if that’s what you’re talking about, except that of course there are more options for git where you just click a button and enter your credentials and away you go. But that’s more a function of the fact that there aren’t as many options for perforce hosting and companies are more likely to host it internally themselves.

If companies are using perforce but aren’t running any CI/CD that’s because they’re lazy/small/don’t see the value.

You said it yourself, there are less options and that's worse. Swarm is a bad joke. Nothing is as integrated as gitlab/GitHub.

Right. Sure I agree with that. You have to do a lot more yourself. I just don't think it's that big a deal though but that's probably just me. Someone running Perforce has probably set that up themselves, and so they vaguely know what they're doing. So if they care about CI/CD they probably have the ability to set it up themselves. Personally I've used CruiseControl.NET, Jenkins, Buildbot, and custom in-house software and the first three support Perforce out of the box. I also don't mind the classic Swarm UI (I don't like the new UI) although I admit I do prefer the GitHub and GitLab UI!

There is the old adage that every bit of friction increases the chance something won't be done.

Swarm feels like a minimal effort to me and just isn't as frictionless as GitHub. Id rather have Swarm than not, but it's not great.

I've been thinking of using git filter to split the huge asset files (that are just internally a collection of assets bundler to 200M-1GB files) into smaller ones. That way when artist modifies one sub-asset in a huge file only the small change is recorded in history. There is an example filter for doing this with zip files.

The above should work. But does git support multiple filters for a file? For example first the above asset split filter and then store the files in LFS which is another filter.

I mean it might work but you'll still get pull time-outs constantly with LFS. It's expensive to wait two or three days before you can start working on a project. Go way for two weeks, it will be a day before you can "pulled" up to date.

I hope this "new" system works but I think Perforce is safe for now.

> People should use the VCS that's appropriate for their project rather than insist on git everywhere.

Disagree. I really like the "de-facto standard" that git has become for open source. It means if I want to understand some new project's source code, there is one less hassle for me to deal with: I don't need to learn any new concepts just to access the source code and all the tooling is already right there.

The situation we have with package managers, dependency managers and package managers for package managers is worse enough. I really don't want a world in which every language or every project also comes with its own version control system and remote repo infrastructure.

A "proper" versioning system doesn't need to be learned since you literally only need a handful of straightforward operations (how do I get the latest version? how do I add a file? how do I commit changes?) - in svn that's 'svn update', 'svn add' and 'svn commit', that's all what's needed to get you through the day, no 'push', no 'staging area', no 'fetch' vs 'pull' and the inevitable merge-vs-rebase discussion... etc etc etc...)

It's only git which has this fractal feature set which requires expert knowledge to untangle.

But if all systems are so similar anyway, why would you need "the right tool for the job"?

If nothing else, you have to install it. There will also be subtle differences between concepts, e.g. git and svn both have versions and branches, but the concepts behave differently. I don't know about Mercurial, but I'm sure they have their own quirks as well.

Also, tooling: I have a VSCode plugin that visualizes the entire graph structure of a git repo really nicely. Right now, I can use that on 99% of all repos to get an overview of the branches, last commits, activity, etc.

If version systems were fragmented, I'd have to look for equivalent tools for every versioning system separately - if they exist at all. More likely, I'd be restricted just to the on-board tools of every system.

> But if all systems are so similar anyway, why would you need "the right tool for the job"?

They’re similar in the UI but the underlying architecture is vastly different, to accomplish different goals - sometimes what you want is an entirely centralized VCS, decentralized VCS, or a mix of both.

As for the tooling, any decent IDE supports different systems equally well. With IntelliJ I can use Git, SVN, and even CVS through the same UI. But yes, VSCode plugin XYZ doesn’t.

In the same way some might be discouraged from contributing to a project because they don't know the language well enough, I've given up on contributing to projects because I couldn't figure out mercurial, and I didn't care enough about the contribution to learn it.

Monocultures can be convenient but are ultimately unhealthy.

To be clear, "fully distributed" also means "can use all of the features offline, and without needing commit access to the central repository".

I can't imagine living without that feature, but I also do a lot of OSS work so I'm probably biased.

How often are you fully offline though? A centralized version system could be updated to work in 'offline mode' by queueing pushed changes locally until internet is available again (and in SVN that would be quite trivial because it keeps the last known state of the repository in a local shadow copy anyway).

> by queueing pushed changes locally

And if you and another developer make conflicting changes while offline? What should happen when you return online?

The same thing as in git: you're not allowed to push your changes to the server until you have resolved the conflicts locally.

E.g. with current svn you get the latest changes from the server, open a diff editor, fix the conflicts and then commit.

The only difference here between svn and git is that svn merges the 'commit' and 'push' operations into one, e.g. instead of not being allowed to push, you're not allowed to commit in svn if there are pending conflicts.

This would be the part that would need to change if svn would get a proper 'offline mode', e.g. commits would need to go into some sort of 'local staging queue' until you get internet access back, and conflict resolutions would need to happen on the commits in that staging queue. But I really doubt if that's worth the hassle because how often are you actually without internet while coding?

It's not terribly common, but more critically it means I can work even when the source forge is down. (And for corporate stuff it means I can work on stuff without connecting to the VPN.)

Also, designing around distribution meant that merges have to be fast and work well -- this is a problem that most centralised systems struggle with because it's not a key part of the workflow. Branching and merging are indispensable tools in version control and I'm shocked how long CVS and SVN survived despite having completely broken systems for one or both. Being able to do both (and do stuff like blames over the entire history) without needing to communicate with a server is something that changes the way you work.

My actual hot take (as a kernel developer) is that email patches are good, actually. I really wish more people were aware of how incredibly powerful they are -- I have yet to see a source forge that can get close to the resiliency and flexibility of email (though SourceHut gets very close). git-send-email has its quirks, but b4 honestly makes it incredibly easy.

(There's also the owning your own data benefits that I think often go overlooked -- reflog and local clones mean that you cannot practically lose data even if the server goes away or you get blocked. I suspect Russian or Iranian developers losing access to their full repo history because of US sanctions wouldn't share your views on centralised services.)

All the time. If you use p4 and git you'll notice the difference just from trying to move clone git repos locally vs trying to make a new workspace.

> People should use the VCS that's appropriate for their project rather than insist on git everywhere.

But git is likely to be appropriate almost everywhere. You won’t just use svn just for big file purposes while git is better for everything else in the same project

The thing is, in a game project, easily 99% of all version controlled data is in large binary files, and text files are by far the minority (at least by size). Yet still people try to use git for version control in game projects just because it is the popular option elsewhere and git is all they know.

> text files are by far the minority (at least by size)

Well yeah because text files are small. Thinking text files are insignificant to games because they are small is a really dumb perspective.

> Yet still people try to use git for version control in game projects just because it is the popular option elsewhere and git is all they know.

Or perhaps it's because it works really well for text files, which are a significant part of most games, and because the tooling is much better than for other VCS's.

> Thinking text files are insignificant to games because they are small is a really dumb perspective.

Fact is that code is only one aspect of a game project, and arguably not the most important. Forcing a programmer-centric workflow on artists and designers is an even dumber perspective ;)

> and because the tooling is much better than for other VCS's

...only for text files. For assets like images, 3d models or audio data it's pretty much a wasteland.

Sure, which is why many studios don't use Git. I was just saying that your argument that code is unimportant because they are text files is super dumb.

Not because it's popular, but because all the tooling (CI/CD) around git is the best.

In games a lot of the tooling assumes P4 so it's often a better choice, on the whole, but if git and LFS was as widely supported in art tooling it would be the clear choice.

> Git is intended to be fully distributed

Which is kinda funny because most people use git through Github or Gitlab, e.g. forcing git back into a centralized model ;)

> People should use the VCS that's appropriate for their project rather than insist on git everywhere.

Indeed, but I think that train has left long ago :)

I had to look it up after I wrote that paragraph about locking, but it looks like file locking is supported in Git (just weird that I need to press a button in the Gitlab UI to lock a file:

https://docs.gitlab.com/user/project/file_lock/

...and here it says it's only supported with git lfs (so still weird af):

https://docs.gitlab.com/topics/git/file_management/#file-loc...

...why not simply 'git lock [file]' and 'git push --locks' like it works with tags?

If you’re making local commits (or local branch, or local merge, etc), you’re leveraging the distributed nature of Git. With SVN all these actions required being done on the server. This meant if you were offline and were thinking of making a risky change you might want to back out of, there was no way on SVN to make an offline commit which you could revert to if you wanted.

Of course if you’re working with others you will want a central Git server you all synchronize local changes with. GitHub is just one of many server options.

I'm pretty sure that if required, SVN could be updated to have an 'offline mode' where more offline operations would be allowed, since it keeps a local shadow copy of the last repository state from the server anyway. But offline mode simply isn't all that necessary anymore. How often is an average programmer in an area without any mobile coverage or other means of internet connectivity?

Shortly after git became popular, the Subversion team released a tool that enabled replicating a SVN repository or any subset to a local independent repository, which could be used for local development and re-synced to the main repository at any time.

I very much dislike git (mostly because software hasn't been "just source" for many decades before git was hobbled together, and git's binary handling is a huge pain for me), but what does a lockfile achieve that a branch -> merge doesn't, practically, and even conceptually?

Binary files can't be meaningfully merged. E.g. if two people create separate branches and work on the same Photoshop file, good luck trying to merge those changes, you can at best pick one version and the other user just wasted a lot of time. A lock communicates to other users "hey I'm currently working on that file, don't touch it" - this also means that lockable files must be write-only all the time unless explicitly locked - so that people don't "accidentially" start working on the file without acquiring a lock first. Apparently (as far as I have gathered so far from googling) git-lfs supports file locking but not vanilla git.

> Binary files can't be meaningfully merged.

I think we really need more development of format-specific diff and merge tools. A lot of binary formats could absolutely be diffed or merged, but you'd need algorithms and maybe UIs specific to that format - there is no "generic" algorithm like for text-based files. (And even there, generic line-wise diffing if often more "good enough" than really good)

I think it would be awesome if we could get "diff/merge servers" analogous to the "language servers" for IDEs some day.

Git actually supports this, believe it or not, although it's a bit wonky (of course):

https://github.com/ewanmellor/git-diff-image/blob/master/REA...

https://zachholman.com/posts/command-line-image-diffs/

Sounds like an infinite rabbit hole :)

The alternative of preventing complex merge situations in the first place through file locking is low-tech, easy to implement, and automatically works on all current and future file formats.

Well, a concrete pain point where I had this problem was Unity scene files (a few years ago). Unity stored not just the assets but also the scene information in binary files, which made integrating that into git an absolute pain. (They made matters worse by also storing "last edit" timestamps in certain files, so git would show changes everywhere even if there weren't any. But that's another topic)

The problem was that the scene information was fundamentally visual (assets arranged in 3D space) so even a diffable text format wouldn't help you much. On the other hand, scenes are large enough that you often would want to work on them in parallel with other people.

I believe their first solution to that was the Asset Server that supported locking. But that still doesn't give two people the ability to work on a scene concurrently.

Eventually, some users went and developed a custom diff/merge tool to solve the problem.

https://discussions.unity.com/t/scene-diff-ease-your-sufferi...

Unity has had a text based scene format available as an option for a long time. Yes it’s complicated looking yaml but it’s diffable and mergeable. They also have a smart yaml merge tool.

An alternative approach (less powerful but simpler) might be to reversibly convert the binary files into a mergeable text-like form before committing them.

I've never done exactly that but I have occasionally decided how information will be represented in a data file with merging in mind.

edit: write-only => read-only of course :)

You should look into how git is architected to support a lot of features SVN doesn't (distributed repos is a big one). When you clone a git repo you clone the full file history. This is trivial for text but can be extremely large for binary files.

Storage is not storage as you can store things as copies or diffs (and a million other ways). For code, diffs are efficient but for binaries, diffs approach double the size of the original files, simply sending/storing the full file is better.

These differences have big effects on how git operates and many design choices assumed diffable text assets only.

If you do a little research, there's plenty of information on the topic.

Reply on news.ycombinator.com