How do game companies share massive files?
bbc.co.uk
bbc.co.uk
If "tech experts" writing news articles aren't really experts, what does that say about all the other experts..?Honestly synchronizing 50GB of contents across the globe doesn't sound that terribly novel or difficult to me, or the article makes a pretty bad job of explaining the difficulty.
50GB is not the total size of the files needing to be transferred - some of the files were 50GB. Knowing the total size would have been interesting for the sake of discussion.
But at any rate the size of the individual files is not what matters. The important part is the total size of the game files and the average change rate. I would be surprised if they weren't able to synchronize everything with a good internet connection and a bunch of rsync scripts.
We use a lot of rsync.
The article title is misleading. They are actually talking about the internal workflow.
Raw assets, art assets, and such, can be pretty large.
When you have a studio in Country A doing PC, a studio in country B doing ps3, all wanting the base raw data but (presumably) transforming it in different ways, between how the build actually happens or artists tweaking something for a platform, it'd become a nightmare pretty quickly if you were to send it across a company private network (often just VPN pipes across the internet itself).
The article is, at it's heart, an ad for a company called 'Panzura', which sounds like an enterprise-specific dropbox. Boxes 'on-premise', security guarantees, and probably pretty expensive.
Most games' source data is much larger than the shipped data size for a number of reasons, mostly because files designed for human editing encode a lot of information not needed by the game itself, and game source data is aggressively compressed making it unsuitable for further editing. A full snapshot of the source data for a game like Battlefield 4 is likely to be on the order of hundreds of GB, maybe getting up to the TB range. And if multiple teams are working on the same source data, then the source data needs to be sent between them.
Also, before the final few months of development, games' runtime data will often significantly exceed the final shipped size because compression and quality tradeoffs have not been decided and applied.
Much like how software doesn't always ship with source code, games don't ship with source assets.
Normal maps are generated from high-polygon meshes that do not ship with the finished product.
Final textures are the result of psd files with layers, reference materials, alternative versions. These do not ship with the finished product.
Maps are compiled from a source format. E.g. vmf -> bsp. The vmf does not ship with the finished product.
http://www.explosion.com/45404/call-duty-ghosts-pc-install-5...
I've been out of game development for a year, but unfortunately git is not being taken up much by large game developers. The main reason cited is actually git's handling of binary files, which leads to every checkout containing many duplicates of huge files. However this is possible to work around and the benefits of git still make it worth it.
The other reason not to use git in game development is because it's incredibly hard to use, and more than half the game development team are not programmers and/or not technical.
Getting them to use good source control discipline is hard enough with TortoiseSVN. I shudder to think the insane mess we would have using git.
As for git being 'hard', yes it's not trivial to learn, but for people using just perforce/svn etc. they end up misusing so much that it's almost worthless - as Linus said, even just sending around tarballs is better.
I think if you're running a team, you need to give your team the benefit of the doubt that they can learn new tools, learn to use them effectively, and be more productive. If you spend some time teaching them why the tools are important and how to use them, then using git isn't really any more difficult than anything else. It's just different.
We create a master repo with all the design assets, statement of work, and any other useful files for the end product. Then we have a client repo as it were of the files we serve in the wamp www directory. Because this is physically different, submodules make little sense. For the WPF builds that all live together in the same directory structure, submodules are perfect. My only hiccup is remembering to commit the subs via the master but that's a habit I need to change. The files in the wamp repo are deployed to client computers using git pull with a read only ssh key so people aren't committing from them.
Our wamp files are primarily text either in straight html or usually PHP. We take care not to commit user editable sections so we don't overwrite what a customer ultimately changes. This is kind of a hand rolled version of the now many git deployment tools out there and it works rather well.
I know this isn't gaming nor are our source files that large but the psds can be astronomical very quickly on just simple web pages. I could only imagine what goes into "next gen games". GIT is still useful for binary deltas but you definitely have to carefully plan how you partition things in repositories. If I can do this, really almost anyone can. Kiln brought excellent binary support to Hg so there's really no need to stick to svn. There may be areas where its still useful but the distributed nature of dvcs is just too compelling when your team is well... distributed. I do hear great things about perforce too and I hope any company uses the better vcs not because they think the users can't cope with anything new, but there is extreme benefit for doing so. Again, if I could help our team move into Git with very little hand holding, you would probably be surprised how quickly people adapted. Especially if something like TortoiseGit isn't that much different than TortoiseSvn or TortoiseHg for that matter.
http://stackoverflow.com/questions/9478023/is-the-git-binary...
Support for chunking of big files would possibly solve this, but the git project is not interested(?).
Of course they do, and I'm ... let's say pretty sure that much of EA runs on http://www.perforce.com/.
Also weird that the article starts with a BF4 image, but then claims that BF4 was developed in Californa. I guess that hurts my national pride (go DICE!).
I still vastly prefer git. Perforce does have some pluses, but it's mostly people just going with what they know.
Programmers always focus on the technology of source control, but the difference it makes is also social. It forces a particular workflow that allows tasks to be carried out concurrently. This workflow could be adopted without necessarily needing git.
That said, I have heard of companies actually couriering semi-large volumes of sensitive data this way.
Indeed some countries and there data protection laws make things most interesting in legal ways to transfer data, even today. No standard on data protection World Wide and in itself a minefield, let alone other industries like medical research companies or those dealing with country wide firewalls.
Sounds more fair that way.
Ahm no, Battlefield 4 is already a PC game and scaled down even on the next gen consoles, so console processing power isnt the limiting factor here.
Why aren't they presenting at more conferences damnit!
also the feedback i see about battlefield's stability and quality has been very poor (on ps4) so I had to laugh when I saw the " to locate defects and improve quality" quote :)