Doom 3 source released | Hacker News Reader

Doom 3 source released

github.com

630 pointspadenot101 comments

    $ sudo perl cloc-1.55.pl --unicode doom3.gpl/
        2014 text files.
        1907 unique files.                                          
         476 files ignored.
    
    http://cloc.sourceforge.net v 1.55  T=40.0 s (36.6 files/s, 22123.3 lines/s)
    -------------------------------------------------------------------------------
    Language                     files          blank        comment           code
    -------------------------------------------------------------------------------
    C++                            517          87051         113094         366423
    C/C++ Header                   615          29640          26891         110991
    C                              170          11407          15566          53520
    Bourne Shell                    36           4529           5476          33717
    m4                              10           1079            232           9025
    HTML                            55            391             76           4142
    Objective C++                    6            709            654           2605
    Perl                            10            523            411           2380
    yacc                             1             95             97            912
    Python                          10            108            182            895
    Objective C                      1            145             20            768
    make                            22            160            253            579
    DOS Batch                        5              0              0             61
    Teamcenter def                   4              3              0             51
    Lisp                             1              5             20             25
    awk                              1              2              1             17
    -------------------------------------------------------------------------------
    SUM:                          1464         135847         162973         586111
    -------------------------------------------------------------------------------

Question 1: Why did you run cloc as root?

Question 2: Why does Doom 3 involve 33,717 lines of bash? That's a lot of bash. I realise it's used heavily for build scripts, but thirty three thousand lines seems excessive.

No clue about question 1 but 32,607 of those lines are all part of curl which seems to be distributed with the source code.

For question 2: It is essentially the configure script of the included curl library.

    $ wc -l neo/curl/configure
    31052 neo/curl/configure

Question 1: Why did you run cloc as root?

Hmmm, because of a really bad habit I need to break. ;)

If your habit is what I think it is, I have the opposite habit - I never use sudo until it tells me I've tried to do something I'm not allowed to do - even if I know it's not going to let me. Then I use "sudo !!"

> Then I use "sudo !!"

Wow. For six years I've been doing "UP + CTRL^A + sudo + SPACE" instead. This is much better.

Off-topic:

I use Ctrl-P Ctrl-A sudo <space> which is exactly the same number of keystrokes as "sudo !!" and a much more general solution in my opinion.

How does it matter?

Not even 1MM SLOC -- that alone is very impressive.

I wonder why the depth fail shadow rendering algorithm had been explicitly excluded from the open source release. It's been around for years now, after all.

There was a patent issue; this delayed the release slightly whilst it was removed.

I give it a week before someone forks and adds it back in, just to spite Creative.

Possibly because it's patented. There was a dispute with Creative Labs about the method.

Would you happen to know if this is Carmack's reverse or something totally unrelated?

Edit: OK. I just googled and it indeed is Carmack's reverse.

It is a patent issue [1].

Check out draw_common.cpp [2] - the patent work-around is contained in the RB_T_Shadow method:

  // patent-free work around
  if ( !external ) {
    // "preload" the stencil buffer with the number of volumes
    // that get clipped by the near or far clip plane
    qglStencilOp( GL_KEEP, tr.stencilDecr, tr.stencilDecr );
    GL_Cull( CT_FRONT_SIDED );
    RB_DrawShadowElementsWithCounters( tri, numIndexes );
    qglStencilOp( GL_KEEP, tr.stencilIncr, tr.stencilIncr );
    GL_Cull( CT_BACK_SIDED );
    RB_DrawShadowElementsWithCounters( tri, numIndexes );
  }

[1] See my previous comment for more info: http://news.ycombinator.com/item?id=2945889

[2] https://github.com/TTimo/doom3.gpl/blob/master/neo/renderer/...

I was under the impression that most large projects end up having to write a custom memory allocator, but after a few random probes into the source I didn't see anything that stood out. Did I just miss it, or did they use the built in one?

Why do you have that impression?

That makes very little sense to do - every single project? Don't you think someone would have released a library to implement the "new and improved allocator" if everyone needs to use it?

That has been borne out by my experience so far too.

There have been several successful efforts at writing better performing general-purpose allocators:

    * TCMalloc (http://goog-perftools.sourceforge.net/doc/tcmalloc.html)
    * jemalloc (http://www.canonware.com/jemalloc/)

There's a great Facebook Engineering blog post about jemalloc which describes why a fast allocator is needed & why the standard malloc is not always sufficient: https://www.facebook.com/notes/facebook-engineering/scalable...

I've worked with several projects where a TCMalloc based allocator (threaded) was used to great success. A simple switch to DLMalloc (non-threaded) can also provide a significant performance boost.

Moderate usage of STL can do nasty, nasty things to memory performance. In one case a game I worked on ran at 5-10fps and a simple, fast change to DLMalloc instantly took perf to 60fps.

It's worth noting that at least on some platforms the system malloc is dlmalloc (presuming that DL is Doug Lea).

Which platforms?

glibc's malloc has long been based on Doug Lea's malloc.

+1 for TCMalloc.

We used TCMalloc at Opera Community[0], and they still might be using it there for all I know ... it greatly improved performance on our MySQL servers!

[0]: http://my.opera.com/

Aren't you using Percona build ?

Not when I worked there, but it's a few years ago. Don't know what they use now.

Most game projects do end up having to write a custom allocator, due to very specific memory budgets, deterministic malloc/free perf requirements, and issues with fragmentation.

No, the point of a custom allocator is writing one that works with your specific requirements. A general allocator will never be more efficient than a custom one designed for the specific task. It becomes just an issue of is the cost worth what the optimization buys you, and games are an area where you need every drop of speed you can get.

There's one here:

https://github.com/TTimo/doom3.gpl/blob/master/neo/idlib/Hea...

Surprisingly there's not a lot of platform-specific cruft in that code. The Xbox version of Doom3 must have been a separate code branch.

the xbox version was done by another activision studio called vicarious visions. no xbox or playstation code can be released due to developer license agreements

Quake 2 didn't have its own memory allocator either. Maybe the system allocator(s) got better with the years.

http://fabiensanglard.net/quake2/index.php

I don't know which Quake 2 you are looking at. Do a code search for Z_TagMalloc and Hunk_Alloc.

To quote the review (which I've read more than the source):

  The Zone Memory Allocator(common.c: Z_Malloc, Z_Free,
  Z_TagMalloc , Z_FreeTags) is still here in Quake2 but
  it is pretty much useless:
  Tagging is never used and allocation/deallocation is
  build on top of malloc and free (I have no idea why id 
  software decided to trust the C Standard Library at this
  point).
  The overflow detector (using the Z_MAGIC contant) is
  never used either.

So, unless I totally misunderstand things, it is half C stdlib, half Hunk allocator?

idlib/Heap.h and idlib/Heap.cpp implement a replacement malloc/free.

From the header comments:

/* ===============================================================================

Memory Management

This is a replacement for the compiler heap code (i.e. "C" malloc() and free() calls). On average 2.5-3.0 times faster than MSVC malloc()/free(). Worst case performance is 1.65 times faster and best case > 70 times.

=============================================================================== */

The source for the physics engine (https://github.com/TTimo/doom3.gpl/tree/master/neo/game/phys...) looks rather interesting. Might contain valuable additions for other physics engines to pick up.

IIRC the source for most of the physics engine has been available for years with the SDK. I think it only including rigid body dynamics and the constraint solver though, no collision code, so it's nice to have that part of it.

I'm not sure where I picked up the info, and I haven't looked at the code to confirm it, but I believe the physics engine implementation is tied tightly to both the collision detection system and the entity update, both of which are quite specific to Doom (in particular, each entity steps physics independently, rather than a whole simulation advancing at once as is common in 'pluggable' physics engines like ODE, Havok etc).

Also, and this is subjective I suppose, I don't remember Doom 3 doing anything that 'felt' or looked better than even what the contemporaneous engines did (such as the version of Havok used in Half-Life 2).

So I guess, there probably aren't valuable additions, but I imagine there is much to tinker with.

For the sake of completeness, here is where I read that: http://www.ode.org/old_list_archives/2004-December/014735.ht...

From looking at the game code, I see odd things like rolling barrels animating their rotation as though the physics engine itself doesn't handle rolling contact.

Any advice on how to dive into reading this codebase?

I've heard the best way to tackle the Quake codebase was to pick a 'component', such as the renderer, and start to understand how that works. Work your way out, one component at a time. Eventually the bigger picture comes together. The same would probably work for this, though unless you're well versed in game and graphics programming nomenclature some of it will go over your head, which is expected - when you see something like that, think of it as an opportunity to learn a new concept and look it up.

My favorite technique is to run the code, place breakpoints in interesting locations and manually type out psuedocode descriptions of each function up the callstacks. That's the fastest way I know to get an understanding of the structure of a new code base.

Barring running the code... I prefer to start with a quick scan through the foundation code. What does the math library look like? How about the file and network I/O? Next, locate the top-level main loop, then the top-level Update Everything/Render Everything functions. At that point, you'll understand the general theme of the code and you'll have spotted lots of interesting avenues to explore.

doxygen

EXTRACT_ALL

Don't waste your time. It doen't have anything ground breaking.

Only took 8 minutes and 22 seconds to compile this, I was quite surprised. Going to have some fun over the next few days that's for sure! Thanks Carmack, heh.

Can you give me some advice how to add the gamedata to the code?I got compile errors for I had no doom3 on my computer(windows 7),I'm downloading it now.

Sure. To compile you only need to get the DirectX SDK (from here http://www.microsoft.com/download/en/details.aspx?displaylan...). Then open Visual Studio (I'm using 2010 Ultimate, but Express should be fine).

(these are the steps that worked for me)

Open doom.sln Go to View -> Property Manager Expand Debug | Win32 Right click "Microsoft.Cpp.Win32.user" and choose Properties Select VC++ Directories

You need to add the following directories C:\Program Files (x86)\Microsoft DirectX SDK June 2010\Include C:\Program Files (x86)\Microsoft DirectX SDK June 2010\Lib\x86

Do the same for Release | Win32.

Doom3 should now compile. Note that 2 projects might fail and 1 could be skipped.

You should find Doom3.exe under doom3.gpl\build\Win32\Release From your Doom3 installation copy the "base" folder, if you have the expansion copy that too.

Doom3 should now launch.

If you have any more problems, feel free to ask!

hi I got his error, on ubuntu 11.04. I moved the base folder to the neo folder but it still didn't work. Is it because it's missing the game data?

  DOOM 1.3.1.1304-debug linux-x86 Nov 24 2011 00:27:04
  found interface lo - loopback
  found interface eth0 - 10.0.0.19/255.255.255.0
  ------ Initializing File System ------
  Current search path:
  /home/xx/.doom3/base
  /home/xx/doom3.gpl/neo/base
  game DLL: 0x0 in pak: 0x0
  Addon pk4s:
  file system initialized.
  --------------------------------------
  ----- Initializing Decls -----
  ------------------------------
  ------- Initializing renderSystem --------
  idRenderSystem::Shutdown()
  Sys_Error: _default material not found

196 clones, 171 watchers, 23 forks; that's just in the first hour from release. Not bad, not bad at all.

EDIT: Doom 1 source code seems dirty when compared to Doom 3.

Skimming over the D3 source, it all looks so very readable. Which is impressive in itself.

A bit off-topic, but I'm curious: why do you think that's impressive? I thought it was pretty commonly accepted these days that readable code saves more time than it costs over the life of a project...?

Because it may be commonly accepted, but it's rarely actually done so well.

Game code specifically is usually the exception to this rule, because it tends to be written to very strict deadlines, ad-hoc requirements, short project lifetime (no need to maintain or support it for years after), and not much expectation of re-use.

This does not all hold for Doom3 as it was also an engine sold to other companies for other games.

A lot of things are generally accepted as "good ideas", but how often are they practiced? Most software management would throw their own mother under a bus if it got the product out the door on anything close to schedule or budget.

Which is to say, code cleanliness is very much not the norm anywhere. And if anything the game industry seems to be a good 10-20 years behind the rest of the software industry in terms of hewing to good practices (clean code, automated tests, etc.)

It will only be a matter of time before somebody converts this to Javascript :)

This seems to be pretty hard wired to build 32 bit binaries only, I'm having a little trouble build on Ubuntu w/ amd64. First you need to install multilibs and 32 bit gcc/g++. But I got stuck when I'd need some other libs in 32 bit versions (zlib, curl and probably openal and others). Ubuntu doesn't seem to ship with 32 bit versions of some libraries as some other distros do (like Arch, for example). Or am I wrong, what packages and what repositories I need to get e.g. zlib 32 bit binaries?

Has anyone had success building this on a 64 bit machine?

Is my only option to set up a virtual machine with a 32 bit distro?

make sure you install g++-multilib so it pulls in the correct version. lib32z1-dev will work for zlib (need to create a symlink since it doesn't look in /usr/lib32). I don't see anything for 32-bit curl though :(

I did install g++-multilib and a few other packages in order to get the build forward. My build finally failed when trying to link the "doom3" executable due to the lack of 32 bit libs for zlib and curl (maybe others too).

I tried to use a very unofficial Ubuntu utility called getlibs (http://frozenfox.freehostia.com/cappy) but that didn't really help, the library search path was wrong and the correct libs were not found.

Fix SConscript.curl to look in /usr/lib32 for zlib (package name: lib32z1-dev ), and to pass -m32 to your compiler.

  /media/d2/sw/doom3.gpl/neo$ git diff
  diff --git a/neo/sys/scons/SConscript.curl b/neo/sys/scons/SConscript.curl
  index 4d889b9..7271dfc 100644
  --- a/neo/sys/scons/SConscript.curl
  +++ b/neo/sys/scons/SConscript.curl
  @@ -14,7 +14,7 @@ class idBuildCurl( scons_utils.idSetupBase ):
   
          def Compile( self, target = None, source = None, env = None ):  
                  self.TrySimpleCommand( 'cd curl ; make clean' )
  -               cmd = 'cd curl ; CC=\'' + env['CC'] + '\' ./configure --enable-shared=no --enable-static=yes --enable-http --enable-ftp --disable-gopher --enable-file --disable-lda
  +               cmd = 'cd curl ; CC=\'' + env['CC'] + '\' CFLAGS=\'-m32\' ./configure --enable-shared=no --enable-static=yes --enable-http --enable-ftp --disable-gopher --  enable-fi
                  if ( self.debug ):
                          cmd += '--enable-debug'
                  else:
  @@ -37,5 +37,5 @@ else:
   
   g_env.Command( target_name, None, Action( build.Compile ) )
   
  -curl_libs = [ target_name, '/usr/lib/libz.a' ]
  +curl_libs = [ target_name, '/usr/lib32/libz.a' ]
   Return( 'curl_libs' )

Linked for me on Ubuntu 11.04 64-bit.

Thank you, this made it link.

I probably would have figured it out by myself, I solve this kinds of things for living. But I was trying to quickly build this before going to work and ran out of time.

There is a fork with better AMD64 support here: https://github.com/dabroz/doom3.gpl

Thank you for pointing this out, once I get the time to sit down on this for a while, I'll have to take a look at all the action on Doom3 in GitHub. Things have really gone crazy, there's literally hundreds of commits on Doom3 in the first 48 hours after the open source release. It's great that they put it on GitHub and not just on an ftp server. This way we can see all the action going on with it.

I can't seem to find references to this anymore, but I seem to remember that the doom3 source code was stolen and released in like '03 or '04. So if that's true then my response to this article is, "Again?".

It was in '02. Id gave alpha code to ATI so they could tune their drivers. One of their employees leaked it.

http://forums.gamershell.com/showthread.php?407-About-Doom-3...

This time it wasn't leaked.

Oh yea. Anyone still has the old code? Would be interesting to make a diff between the alpha and this one. Not to see specific line-by-line fixes but more just to get a general impression of how feature complete it was back then.

The difference is that this is an offical release and is under GPL. So you can fork this code and sell the results.

I'm impressed by the cleanliness of the code base, very readable. From my experience, this is not the case with many C++ open source projects.

...That's because Doom 3's engine is not an "open source project" as we know them ;)

Nor was it closed source as we knew it. I wish I got paid to work on code that nice...

How do I compile the map editor in linux? I guess it's in neo/tools/radiant but I haven't found a way to build it.

You don't. It's only implemented for Win32 (see neo/tools/edit_stub.cpp).

git clone https://github.com/TTimo/doom3.gpl.git

and just browse around locally in your favourite editor.

Tip: they added direct URL cloning a while ago, so you can copy/paste without adding .git:

    git clone https://github.com/TTimo/doom3.gpl

Given the size of the codebase, you most probably want to use ctags, or something like Intellisense for Windows.

Anyone got this error?

  DOOM 1.3.1.1304-debug linux-x86 Nov 24 2011 00:27:04
  found interface lo - loopback
  found interface eth0 - 10.0.0.19/255.255.255.0
  ------ Initializing File System ------
  Current search path:
  /home/xx/.doom3/base
  /home/xx/doom3.gpl/neo/base
  game DLL: 0x0 in pak: 0x0
  Addon pk4s:
  file system initialized.
  --------------------------------------
  ----- Initializing Decls -----
  ------------------------------
  ------- Initializing renderSystem --------
  idRenderSystem::Shutdown()
  Sys_Error: _default material not found

Hmm... Looks like this is an Xcode 3 project file. I can still get Xcode 3 from Apple's developer downloads, but it looks like it only includes the older iOS SDKs, not the OS X 10.5 SDK required to build this.

Try: http://stackoverflow.com/questions/5378518/how-to-add-base-s...

It's kind of sad to see the expressions of surprise that it actually builds and runs. Implying that most source code releases fail test case #1.

In my experience, it is usually an environmental assumption that is mismanaged rather than the "source code" itself being broken.

Whether that is failure to check for a dependency, failure to document a dependency or failure to clearly specify which specific version of a dependency is required. And the pain of autotools is a completely separate chapter in describing how "my" environment can differ from "theirs."

All of that just goes to show that state management is _the_ hardest part of software.

True. Generally the failure of the packager is that they never do a clean install and build. And yes that is tedious but it finds so many issues, and isn't so bad with VMs.

I am getting the error, "scons: * [build/debug/core/sys/scons/doom] Source `/usr/lib/libz.a' not found, needed by target `build/debug/core/sys/scons/doom'. scons: building terminated because of errors." Anyone knows what I should install?

zlib

or zlib-dev if your distribution separates out development packages

Anyone else having trouble trying to run it? got the assets in place, get to the menu fine, hit new game then get a black screen with an empty box in the middle. Suspect its the copy protection?

Are there any parts of the quake source code in here?

Heard that the experimental render path with more advanced features is included in the source.

Can anyone here confirm/deny that?

Does that mean we don't have to pay to get the game now? Compile the source and playable?

Nope, it's only the engine. If it's anything like the other iD software code releases, you will be able to use the content from the game disc with the engine built from source, or you will be able to extract the content from the demo if you dont have the game.

No. “This source release does not contain any game data, the game data is still covered by the original EULA and must be obeyed as usual.”

You should be able to play a total conversion that does not rely on the original game’s assets, however.

Suggestions? The Dark Mod looks interesting.

http://www.thedarkmod.com/

Requires Doom 3 game data to play. It uses many Doom 3 resources, and other resources are derived from retail game data.

Quake 1 and 2 had demo versions and you could use the game data from there. Is there such a thing for Doom 3 ?

thanks for this. i haven't even fully gone through quake 3's source yet.

I'm still on Doom :/

Installed Doom3 using the CDs and updated to the latest release. Then opened the sln file in VS 2010 and changed the command line args for debugging to point to the data of the game installed from the cd (instead of steam).

Now what is the matter with default.cfg and what about "Unknown command 'vid_restart'"? I am not familiar with the structure of idgames.

  ------ Initializing File System ------
  Current search path:
  C:\Program Files (x86)\DOOM 3\base/base
  game DLL: 0x0 in pak: 0x0
  Addon pk4s:
  file system initialized.
  --------------------------------------
  Unknown command 'vid_restart'
  idRenderSystem::Shutdown()
  Shutting down OpenGL subsystem
  ...shutting down QGL
  Couldn't load default.cfg

ah.. it works w/o the base at the end! So the correct path is "C:\Program Files (x86)\DOOM 3"

I got a similar error on linux. How did you get it to work?

I hope this means that there will be an actual Doom package for Debian in the near future.

Any gambling types fancy a little wager? Which will come first, a stable JVM port of doom 3, or a stable build of Textmate 2? The only problem is, I really can't think which side I'd come down on.

NOTE: For those primed to accuse me of hating: this is as much a compliment to certain hackers and their fixated attitude towards porting iD code dumps, as it is a snide reference to textmate2's status as vaporware.

Reply on news.ycombinator.com