Ask HN: How do game companies keep their files organized for a large-scale game?

The best way to do it that I've found is to arrange a folder hierarchy of "where" rather than "what." So there's no one place with all the game's rock models, but rather firelands/towerofeternalflame/courtyard/rock01.3ds and so forth. "Where" doesn't have to mean a literal location; it could be playerbases/horde/rock.3ds or something. Though for most games, everything not tied to the player really can just be its literal location. It's crucial that these directories have a mix of models, textures, and whatever else you need, instead of a mass of parallel structures.

If there are automated tools, they should run on the entire asset directory and dump to a new directory in whatever hierarchy makes sense to the tool. When that turns out to be slow, check "file modified" timestamps in the tool instead of dumping required knowledge on the artist.

Getting artists to actually use source code control instead of having personal folders full of things like rock01_retouched_new_final_final.3ds is a bit of a pipedream, but you can at least put the asset directory under source control and make that the only way to see things in the engine.

Getting artists to actually use source code control instead of having personal folders full of things like rock01_retouched_new_final_final.3ds is a bit of a pipedream

BS. Just hire better artists. It sounds like hand-waving, but I've worked with both kinds of artists. You're building games, so if you focus on only hiring artists based on their portfolio you might end up with a bunch of stubborn artists who need to be hand-held and worked around constantly. If instead you focus on hiring artists who are good at art AND who can demonstrate a propensity or desire to learn new workflows and and to be flexible, then you'll end up with a bunch of team players.

This is a common complaint about artists, but it has more to do with culture and expectations than anything else.

Agreed. While a few artist keep a few _01 _02 files around, mostly as backup/undo files, my last few years in the industry, they all used some kind of version control (maybe not git, but perforce and svn with their visual tools)

Sounds like perforce is common when there are lots of binary files - is this mostly due to storage concerns?

Bunch of factors:

  - Really good binary performance(light years ahead of git).
  - Proper locking for files that can't be merged(basically everything in the art dir).
  - Proxy support for co-located teams.

Game repos can easily approach multiple TB just for the art assets. In that area Perforce is once of the few VCS that scales appropriately.

+1. Also easy graphical interface - P4Win/P4V, and easy to understand the system of changelists. It also helps that older changelists mean things submitted in the past, newer in the future - e.g. in contrast with digest/fingerprint schemes like other systems use.

Yup forgot about that but a good point as well.

Yes, perforce is really good with large binary files. Only recently Git LFS allowed for git repositories to store large binaries as well.

We use git with lfs for our game. It has on occasion been a bit of a pain, however perforce is too...unfortunately there isn't a really great solution atm, just a bunch of more or less flawed ones.

Can you comment on the problems you ran into with LFS?

sometimes it/git would mess up for unknown reasons and people will get the text file with a pointer to the lfs-hosted asset instead of the asset itself.

Also it's very slow for small files, and the filtering is according to file extension.

So e.g. if I have some massive mp3s for (sometimes many minutes of) music but also many (100s-1000s) small ones for sound effects downloading 1000 small mp3s take a couple seconds per file even if each is only ~40kb big (normally download rate is a couple MB/sec).

Not quite games but back when I was doing Flash cartoons in the 90s we had an elaborate scheme for naming art assets. It was something like:

3-4-dd-rhand-07 - episode 3, scene 4, dirty dog, right hand, from keyframe #7

(there were naming conventions for background elements too but they were somewhat looser IIRC)

We had a cheat sheet for this with standard abbreviations for all the recurring characters, and a list of body part names. Everyone had a copy stuck to the wall or floating around their desk. I think that was an important part of what made it work. Don't expect new hires to pick it up by looking at the results of the naming scheme; codify it into a document you hand to them as part of orientation.

(We also accessed a lot of this stuff visually; the file handed to an animator would have all of the keyframe drawings in each scene laid out in sequence, with the raw art separated out into individual parts and turned into named symbols right there on the working canvas.)

All of this was something we hashed out over time; early stuff had a horrible mishmash of naming schemes. I'd advocate having a discussion about Organization as soon as possible, agreeing on a standard and sticking with it for new assets. Get the old stuff reorganized when you have a chance, maybe throw that task to an intern if you have one.

    Code
    	Game
    		UI
    		Gameplay
    		GFX
    		Physics
    	Editor
    	ThirdParty
    SourceAssets
    	Levels
    		Fire
    		Ice
    		Swamp
    		Volcano
    		Sewer
    	Characters
    		Player
    			Models
    				playerInital.fbx
    				playerAwesome.fbx
    				playerGodlike.fbx
    			Animations
    			Materials
    		Merchant
    		Thug
    		ThugFat
    		ThugTall
    	Objects
    		Urban
    		Indoor
    		Outdoor
    	UI
    		Menus
    			Boot
    			Main
    			Options
    		Icons
    ProcessedAssets
    	Pak1
    		(UI stuff)
    	Pak2
    		(Level 1 stuff)

Everything goes in source control. Directories are organized to group together similar work. Names are a mixture of searchability for the artists and conventions needed by the code.

They don't. At least artists are really bad at it when naming their files. I've had to constantly rename assets received from artists to give them some sort of consistency, even after telling them how to name it.

But yeah, Perforce version control is pretty important. It's almost designed to help manage assets, especially large ones. And it's free for under 20 users, I think.

Beyond that, you might want to include the level, screen, and/or date for the assets in your naming structure. Pretty much every company, and sometimes every project, uses a different structure, though.

I worked on a commercial MMORPG project for a while. The game assets were hosted on an unversioned asset server instead of in Perforce. The same asset server was shared by every branch and release of the game client, which made backward and forward compatibility a major hassle. None of the artists understood what the big deal was. I feel like I should write a post-mortem of that project some day.. :)

Kotaku has a great post-mortem on the development of Bungie's Destiny game, including fantastic horror stories about the design tools:

  “Let’s say a designer wants to go in and move a resource node two inches,” said
  one person familiar with the engine. “They go into the editor. First they have 
  to load their map overnight. It takes eight hours to input their map overnight. 
  They get [into the office] in the morning. If their importer didn’t fail, they 
  open the map. It takes about 20 minutes to open. They go in and they move that 
  node two feet. And then they’d do a 15-20 minute compile. Just to do a 
  half-second change.”

http://kotaku.com/the-messy-true-story-behind-the-making-of-...

I don't know how much I trust that source. Maybe they were talking about lightmap baking but any system that incurs more than a 2 minute cost for testing a tweak is going to kill any chance build a meaningful game.

Game design is built upon rapid iteration and anything in the way of that will kill your title.

Have you played Destiny? :D

Now I understand why players complain about Destiny being too slow with adding any new features.

I've used Perforce at multiple companies, but Subversion is a great alternative today. A lot of people's Perforce-vs-Subversion biases were formed years ago, and since then (as a result of pressure from git and all the other alternatives available today) Subversion has added a lot of really great features.

PlasticSCM is a newish alternative. I can't vouch for it, but it looks really good. It tries to give you the best of Perforce and DVCS in one package. It also cleanly solves some problems that are hard to solve in Perforce (e.g. X-links https://www.plasticscm.com/documentation/xlinks/plastic-scm-...).

Does subversion have good proxy support? It's been years since I touched it but that was one thing that saved our bacon with Perforce and distributed teams.

We have tried out plastic but lost our changes on a few occusions because the tool came to a state where you simply couldnt merge. Stashing in those cases also failed. It was terrible. Losing your changes should never ever happen especially for a quite pricy solution.

We are now using git lfs which so far works like a charm.

when I was an intern at microsoft, I worked on a tool lionhead had built for artists, which used perforce to store different versions of art assets.

It was incredibly helpful. Lionhead had some good tech going on to solve this exact problem OP describes.

First, you need to distinguish between in-game assets, and the source models in Maya, Photoshop, etc.

For the latter, most large games use an asset management system. Either homebrew, or something like Alienbrain. (Well, people used Alienbrain when I left. Not sure what's hip these days)

For in-game, there's usually a fairly strict organizational scheme. It depends on the game, but usually folders are about "location" - which map, which part of the world, which episode, etc. Asset names indicate a name, and various other bits like left/right hand hold, color, or whatever matters for that asset.

It's usually the lead artists job to come up with said scheme, unless you have a tech lead for the asset pipeline.

If you look for assets, you do a search in the asset management system, which also contains the in-game name and location.

For smaller games, you make do by manually organizing, but as things grow, you'll start hand-rolling tools.

Organizing assets is hard because the dependencies are complex and change often. We solved this at a previous company using in-game editor tools (material editors, level editors, character editors, AI script editors, etc) that all interfaced with a database back-end. The database didn't store the files, those were stored in Perforce (which the tools interfaced with as well). The database just stored metadata for each version of each asset along with the relational links between assets. So, for example, a particular 3D model referenced a particular material (storing just its id/foreign-key in the DB), which in turn referenced a set of textures and perhaps a shader.

An asset browser allowed you to perform queries (e.g. show me all the characters from level X which reference materials which reference texture Y). We had tags/labels, comments, everything you could want.

When you do it like this your folder organization becomes irrelevant. You can store everything in a single directory with a GUID for a filename if you want. Or store all textures in one place and all 3d models in another. The down side is it's quite opaque, everything must be done within your toolset, so you need great tool support.

It's hard. I personally believe that designing large scale video games is one of the most complicated kinds of programming projects out there - so don't try to reinvent the wheel.

Like daodedickinson recommends; you want to look at how AAA game designers design their systems, then imitate them. Your best avenue to figuring out how games are designed is by understanding how game modding is carried out in AAA games. In many games, almost all of the content and mechanics within the game are defined using the same scripting interface that mods are created through.

I recommend using games like Skyrim, Morrowind, Mount and Blade, and The Witcher (i forgot which one shipped with all the game's scripts in the install folder, might have been 2).

> Not all rocks are the same, so I am pretty sure they didn't name it "rock01" "rock02"

I wouldn't be so sure...

Also 'rock01_final', 'rock01_final_final', 'rock01_new_colors1', 'rock01_new_colors2', 'shirt_long_AC07D4' (different item textures using hex color code as an id number), 'rock01_george' (for george's version), etc. All the bad naming you can imagine I've seen with asset naming.

Does anyone else see the similarity between designing Redis keys and game art asset filenames?

Being able to mine as much metadata from a file name and path is a plus. As is having a logical naming convention that can be queried from the string programmatically. Not to mention fast lookup and asset name generation merely from string concatenation...

This is a good problem to have, mfcecilia! You are on the path to making great games. Its stuff like this that gives you the confidence to say my solution is just as good as anyone else's and I am just going to do it and learn from my mistakes. Time pressure is actually your best ally. If you want your game to be playable by the end of the decade you will solve this challenge and level up to the next big one. Such as figuring out how to organize your assets in a single compressible pak that is laid out to support fast and efficient patching via binary diffs!

Good luck! And if you are still stuck check out the Unity and Unreal engine docs and forums for further tips on pipeline production. And of course, post a link when you build a playable alpha ;)

> Does anyone else see the similarity between designing Redis keys and game art asset filenames?

I always thought Redis keys use filesystem-like names just for ease of use. Pretty sure you can name them whatever you want though, using colons or dots as separators if you don't want to use slashes. But yeah, I do see your point. Think it's just the most straightforward way to name something honestly.

You'll never get around the source of the problem, which is that it's not the priority of the artist to figure out how to name and organize things. It's the same as bothering developers to "update the wiki". Pick a convention and structure the culture around that by providing a spreadsheet of assets or an equivalent document. Maybe the artists "own" their assets, so you lead with their username. Maybe you're building the environments collaboratively, so you lead with the environment first. Maybe you envision the project being a long-term thing like an MMO, so you start with the year and month. These are things that a good producer will keep their eye on, because they reflect how the team is communicating.

The most important thing, from the technical standpoint, is that your asset pipeline is robust and can handle weird things. The engine shouldn't consume files haphazardly, it should deal with "built" assets, where you can trace things from the in-game content back to the source file. A configuration layer usually appears to give an in-engine ID to a loaded asset, because files and assets do not map 1-to-1. You can choose to fight the organization battle in that layer, rather than in the file system. When you have a slick IDE like Unity, this is all built into the GUI, but a custom engine can just use a batch process to do the same.

Version control is useful - not a DVCS, because it doesn't scale to binary assets, but SVN or Perforce. Artists will groan about waiting around for the VCS to resolve things and the inevitable problems with locking, renaming, and deleting stuff, but their lives will be better off overall if it's in their workflow loop. It's "10 hours waiting on SVN Updates" vs "30 hours debugging project history".

Here's an article on "import" vs "export" based pipelines, where the thing being "imported" to the engine is a common, standardized format that every tool can export - rather than one proprietary tool the team happens to use:

http://people.cs.aau.dk/~bt/GameProgrammingE09/Pipeline.pdf

Perforce + Diligent lead tech artist.

Also most of what you see in the final game output is packaged and minified to a high degree in order to improve performance/memory/etc.

Usually the lead tech artist just sets a standard and everyone conforms to that standard. Art assets(.psd, .ma, etc) are usually stored in a separate section than in-game assets.

Skyrim's dev tools (the mod kit) should give you a hint. At least I know with Morrowind you could get a good idea of their naming conventions.

You can get a snapshot of how big game studios organise their art assets by using various modding or decompilation tools to have a look inside their package files.

For example, you can use MPQ Explorer to look inside the package files of World of Warcraft, and see how they organise their tens of thousands of assets.

Generally, as others have said, it's by "where" rather than "what", or sometimes "what"/"where".

Beyond that - Perforce is the usual asset control system for these things. It offers the ability to tag files, which makes searching easier. When you're dealing with a huge pile of art, searching is basically the only way to go, so good tagging is very important.

(Source: I'm not a game designer, but I've worked with the complete art libraries of multiple triple-A games over the last two decades creating Machinima films, either working directly with the studio or on a modding basis. I know far more about the internal file structure of WoW, for example, than is probably healthy.)

I think there's a difference between the files layout on a disc/install folder and a repository. The layout on the disk can be optimized for loading (even duplicating the assets sometimes). OP is asking for a repository hierarchy I think, since he's talking about sources files too.

Oh, yeah, there definitely is.

OP, my bad if you were specifically looking for repo hierarchies. That's not the side of things I'm usually involved in :)

This may seem obvious, but some people just don't get it, and it can be easily taught: I like to purposefully and consistently use "big-endian" names for assets, classes, functions, variables, events, files, directory structures, etc.

The desired result is that related names are convenient grouped together in an alphabetical sort.

A rule of thumb is: the most important part is at the beginning, and most specific part is at the end.

But what is convenient, important or specific is context sensitive and debatable, since there are many ways to name the same thing in different kinds of hierarchies.

Sometimes it's ambiguous what's most important, like when you're classifying things along two or more dimensions, like x y and time.

Should it be (MouseXCurrent, MouseXLast, MouseYCurrent, MouseYLast) or (MouseCurrentX, MouseCurrentY, MouseLastX, MouseLastY)?

In that case I'd put the most tightly bound dimension last (x and y, which are usually used together as a 2d vector, so keep them adjacent).

id software's were usually not too bad:

Quake's pak0.pak: https://quakewiki.org/wiki/pak0.pak

Quake 3 Arena's pak0.pk3: http://openarena.wikia.com/wiki/Pak0.pk3

But that's because John Carmack et al. were geniuses that were ahead of their time, who put effort into making fun out of programming games.

It became especially important as people started buying Quake's engine and modding it to be different games entirely.

https://en.wikipedia.org/wiki/Game_engine#/media/File:Quake_... for example

Pak files aren't really relevant here. At build time or before it, assets will be built into pak files, but before that (as they're being created and edited) they live in a directory structure and that directory structure is committed to source control.

From what I've seen, and as you guessed, each game handles it differently and no matter what conventions you have, someone will be grumpy. But if you're just looking for inspiration, I would look at the content for UE4/UnrealTournament which is about as good as it gets for 'industry standard'. It's free to get access to and you should be able to get some ideas by poking around.

The smart ones have asset management systems, big databases that track what gets used where how.

The Scheme hacker Shiro Kawai has, as one of his claims to fame, having written an asset management system for Squaresoft (now Square Enix) -- not for any of their games but for the movie Final Fantasy: The Spirits Within.

What you are looking at is a classic problem in all of programming: making data machine-readable vs human-readable. Relying on file names for any sort of data is bad practice, IMO. This info should be contained within metadata within the file. So in the file system "object_0001.obj" might contain metadata like "label" "tags" etc - that said you may want two versions of the data - let artists organize their files according to a human-friendly system, but when they are imported into the engine the appropriate metadata must be filled out and the file gets automatically saved to a specific location determined by the engine or editor.

Doesn't that make it pain when debugging or heading down to the directory level? I think rememberable and commonsense file naming makes sense.

It totally depends on the scope of your project, IMO. If you are a one man team making a small indie game, then it probably makes the most sense just to use a simple file system with the naming convention you suggest. The reason I would avoid such a thing in a bigger project is that inevitably such a system will break or become complex if it is not managed by a machine (even I, as a one man team, have hit this problem and I had to go in and rename hundreds of files for a simple change). A simple analogy is an SQL database - you could depend on naming stuff precisely by certain key values to make sense of it, but it becomes more powerful if you provide it with some simple data for each entry and let the computer make sense of it (select all objects with tag "static" and "rock" and "desert" for example). A given convention will force organization inherently. Every time you need to update an object you simply look it up in the editor/engine and reload it from a file you select.

Well, in quite a few cases they don't. For example, I've seen the file structures for quite a few 3DS games via the modding community, and the files are often just named complete gibberish. Like 'file1music.brr' to 'file2639music.brr'.

But from games which do have somewhat logical structures, it's usually that the files are seperated by purpose. So the music and sound effects in one folder, the textures in another, some level files in another folder, etc. And the names would be somewhat descriptive of the content, like 'course_desert_1.lvl' If you're lucky, this is also backed up in a git repository somewhere and has version control.

There is a cultural problem in game development where they think the performance imperative makes them the epitome of software development. They do not understand how anyone in any other field of software development could ever touch them in terms of pure, unadulterated, programming prowess. As a result, tools, lessons, and best-practices learned in other fields are nearly completely ignored. Not-invented-here syndrome is strong.

There is nothing particularly special about game development. Yes, there is an emphasis on real-time operation that changes the specific design of applications. But there is no reason to believe that should have any impact on project management.

Well, there is one specific thing not common to other types of programming: large amount of small-to-large binary files to handle. Most of those files are in proprietary formats which makes it hard to automate stuff. I believe OP was asking exactly about managing these files.

I have to agree with the rest you said. Discussions I had with game devs show that most of them are clueless about databases, web technologies (aside from HTML5 JavaScript) and tend to invent their own sub-optimal solutions. What's the real problem is that some of them share those and get adopted as standard in the gamedev community - because they don't know any better.

They mostly don't: https://www.facebook.com/10153270300786451

Assuming Perforce, or perforce like system. Couple of things.

//depot/main/game - all your code + exported (but not yet converted) data lives there. You can give this to QA, Build manager, scripter, coder and they can build a level, compile the game, tools, etc. if needs so. By exported here is what I mean: - Exported textures, models, sound, etc data ready to be further on converted to platform specific formats. The original data was kept in //depot/source

Keep several department oriented //depot/source p4 views. Each specific department would get mapped this in their view, others would not.

//depot/source/sound - For example big huge 24-bit sound files + whatever other extras ProTools or other software writes. Suitable for sound folks to keep their project settings, experiments, settings, but not suitable for everyone else to see.

//depot/source/textures - Your big ass huge PSD, or whatever else source files - containing all layers that you've built your textures, or whatever else a texture/material department might see fit (there might be more than one - character, environment, etc. departments with different organizations). Not suitable for the main team, great for whoever works on textures.

//depot/source/animations - For example big huge FBX MotionBuilder or something else files. Suitable for animation people to edit, but not suitable for the game tools to read them and convert them every time. //depot/source/models - etc. - your big ass again .3DS, .MA/.MB, even .FBX files. Great for modelers, not great for everyone.

//depot/source/something-else

You might even have external people working for you, you don't put their stuff directly with your stuff (legal maters, payment, review process, etc.)

//depot/outsource/BlamStudio/textures

//depot/outsource/BlamStudio/models

//depot/outsource/CoreySounds/sound

//depot/outsource/OrchestraYou/music

You also don't want them to have access to everything you do, and you need to establish way to review their assets and how to communicate work - what needs to be done, requirments, etc.

.....................

So a typical game developer would only need

//depot/main/game

some studios would even submit back converted data, for example:

//depot/build/game/data/ps4

//depot/build/game/data/xbone

//depot/build/game/data/wiiu

//depot/build/game/data/pc

others prefer to convert while building. For example all huge ass PSD files with lots of layers were probably exported as .TIFF, .PNG, even .JPG for insanely big backdrops, and from then on these would get converted to swizzled optimized textures - for example on all iPhone/iPads you get PowerVR chip, and it has PVRTC texture compression, unlike ETC, DXT for PC.

It all depends also on your team.

So a WiiU only programmer, might just sync

# Not ideal locations, but something that might work:

//depot/main/game/... c:/blah/game/

//depot/build/game/data/wiiu/... c:/blah/game/build/data/wiiu

Desktop/Bob's Files/New Project/Misc/Stuff for Jim/Important First Draft FINAL v2(1).doc