Hacker News Reader

Comment by patio11 | Hacker News Reader

I got the hero speech too, once. If anyone ever mentions the word "heroic" again and there isn't a burning building involved, I will start looking for new employment immediately. It seems that in our industry it is universally a code word for "We're about to exploit you because the project is understaffed and under budgeted for time and that is exactly as we planned it so you'd better cowboy up."

Maybe it is different if you're writing Quake, but I guarantee you the 43rd best selling game that year also had programmers "encouraged onwards" by tales of the glory that awaited after the death march.

View on HN

3 guys wrote the world's first game that rendered a truly three-dimensional world, and they did it in software, and it ran at 20+ frames per second on my Pentium 75. It's not quite correct to talk about Quake the way you would talk about almost any other software project.

BTW, if you're curious, Abrash goes into greater depth at the end of his "Black Book of Graphics Programming", which I think is now freely available online.

Don't forget Wolfenstein 3D. While not directly germain to the topic at hand, this makes Quake second and probably more likely its development was handled like any other software project.

I think you misunderstood the parent post. Wolf3D was not really 3D, nor was Doom after it, and many other FPSs that came around at that time. Quake was the first really 3D FPS (e.g. you could walk under a bridge without "tricks").

I don't know what development process they had at ID at the time, but given that it was a Carmack project, I very much doubt it was anything like "any other" software project.

Descent came out over a year before Quake and was full 3D (except the annoying hostage sprites). It may have taken advantage of being set in a mine to have twisty passages that obstructed views, but it did have its fair share of wide open spaces (most reactor rooms, for instance). I think the reason Quake gets all the credit is due to its popularity, not for being the first.

There was also Driller in 1987 which had full 3D representations though you could rarely take advantage of it. It was based on the Freescape technology which I played with in 3D Construction Kit in 1991 and that was definitely full 3D in terms of movement (though not a game in itself).

Was this similar to Oblivion? Where you had to plant some gas valves to relieve the pressure on the planet? Awesome, frustrating little game :)

Descent used affine texture projection, which makes texels slide around as your viewpoint changes, unless extremely small polygons are used (they weren't). It also had extremely simple lighting (intensities set at vertices, interpolated in a simple gradient in screen space like the texture projection). Neither of these things means you're wrong in calling it "full 3D", of course. I don't recall if the topology allowed room-internal objects other than the robots; I think it did.

I'm pretty sure a few of the LookingGlass games came out before Quake and would qualify as 3D.

Ultima Underworld came out in 1992: http://en.wikipedia.org/wiki/Ultima_Underworld

> Quake was the first really 3D FPS (e.g. you could walk under a bridge without "tricks").

Sometimes the tricks were cool.

In Bungie's Marathon for instance, the map format allowed you to have multiple rooms occupying the same physical 3-d space. Imagine a spiral stair case that doesn't go up or down, you just run around and around this long corridor like a manifold mobius loop. Quite fun in multiplayer maps with radar. Also worth noting that Marathon was the first game that allowed rocket jumping due to being able to look up and down slightly.

Obviously Marathon had nothing on Quake, but it was really a great multiplayer FPS for its day.

I also remember Mechwarrior 2 came out before Quake and was mostly 3D-ish. Granted most of the levels were flat but you could jump (with jump jets) and climb on top of some terrain.

Abrash's Graphics Programming Black Book was my favorite programming book in high school. I read it everywhere I went, even when camping. I highly recommend it to anyone who wants to learn how the lowest-level graphics primitives were drawn before GPUs. The first half of the book also makes for an excellent guide to optimizing algorithms by reducing overhead and converting to assembly language.

My only complaint about the book was that the publisher (Coriolis?) included a notice at the front saying that no code samples, algorithms, etc. could be used in any project without their express approval, but most of the algorithms and code were previously published elsewhere.

I used to do graphic demos in Pascal, in 286 mode, using hand-coded procedures in assembler for the graphics parts, using Mode X for buffering the animations, like 320x240 resolution, 256 colors and only one virtual screen (although more were possible).

Doing 3D was pretty cool too - I couldn't do fancy stuff, but I could display and rotate 3D objects, drawing only visible polygons, draw textures and even have simple lighting.

To me it seems weird how kids work these days. I don't know what a pixel shader is for instance.

To me it seems weird how kids work these days. I don't know what a pixel shader is for instance.

Thanks for bringing up some nostalgia. A pixel shader is roughly analogous to a hand-optimized texturing routine, except it runs on the GPU.

I think games are fundamentally different than web apps in a couple of ways.

(1) There tends to be times where having the product ready is much more profitable than other times (holiday season)

(2) It's a lot harder to rev games.

The end result is you often end up on death march slogs. And unfortnately the state of project management is that they're surprisingly hard to avoid.

Part of the reason is human nature. If we honestly schedule everything out and see it will take 12 months, it will take longer. Because at month 8 it will look like we have three weeks of work left -- to everyone but the project manager and a senior dev or two -- who realize that there is still an honest to goodness four months of work left. On paper it doesn't look like four months of work, but it is (especially because a lot of things are fixing bugs to things that you don't know yet are broken).

So what happens is the pace dev starts to slow down. People start taking vacation, or three day weekends. Before you know it, its month 11 and you're further behind than you were at month 8! WTF!?

You can't just ship the game with this quality. Characters disappearing, levels not loading every other time you play them, etc... A web app you could, and just fix it over the next three months. With a game, while you can service it, its a lot harder, and day 1 reviews are super important -- a 70% metacritic score means no one will even care that you serviced it.

I'm not saying death marches are a good thing. But I've seen that in certain industries they do seem harder to avoid.

"I'm not saying death marches are a good thing. But I've seen that in certain industries they do seem harder to avoid."

How hard they are to avoid depends entirely on how good the management is at coming up with excuses for them and at using carrots and sticks, and on how gullible and desperate their developers are.

I'm not sure that's true. But if you look at certain industries, like web startups, death marches are almost the complete culture.

Try to be employee #1 of a YC startup while being upfront that you work 40 hours a week, period. Not going to happen in many cases. The death march starts on day one in this industry.

"in many cases" being the operative term there.

How many YC startups would kill to get Steve Jobs to work even 1 hour a week for them as employee #1?

It's all about perceived value and negotiation. If your company values you enough, they will not make you work unreasonable hours. And if a non-workaholic employee values himself enough, he will not agree to work unreasonable hours.

Death marches are definitely a pathology in the computer industry, as they are in the medical field, where residents are forced to work insanely long shifts without sleep. In both cases, people's mental and physical well being is put at risk, the chance of burnout increases, and the quality of the results suffers.

It does not have to be this way. What will it take for management to stop understaffing and overworking their employees? And how long will employees consent to being worked in to an early grave?

Companies pulling this sort of crap is what really makes me wish the computer field had some effective unions that could collectively bargain for reasonable hours for reasonable pay. I know I'd join in a heartbeat.

It does not have to be this way.

It doesn't, but it will be. :-)

I worked on bug, almost full time, about a decade ago, for about a month. A single bug. I hadn't anticipated the bug, and didn't plan for it. I've seen teams spend months trying to hit perf targets. It's the type of thing that isn't uncommon in our industry. I'm not sure how you plan for it.

What will it take for management to stop understaffing and overworking their employees?

Many tmies its not understaffing. Again, I've seen teams struggle to hit perf targets. If you doubled the size of the team it wouldn't make their lives much easier. It may even make it worse.

Our industry is one where you are fundamentally solving new problems (because if someone else solved it, you'd use their code).

Now maybe you're saying we need to toss schedules altogether. Things are done when they're done -- and no crunch time. You do it with 40 hours/week for however long it takes. I'd love to see that dream happen to, but I honestly don't think it ever will.

I know where you're coming from, and what you're saying is currently true, I do however fundamentally believe that the death march is not a given. It is a choice that the games industry has made because it:

a) Doesn't know any better

b) Doesn't care enough to change (there are always hungry young fresh graduates to replace those old guys who keep complaining about not seeing their children grow up)

It's partly a process thing, and it's partly a technology thing. I'm researching game testing [1]. Perhaps if we can get game testing to be better, we might end up with better processes, such as how TDD helps other software domains. Same goes for other technologies leading to other better processes.

Lots of really complicated software gets made every day, but only the games industry really uses terms like "crunch time" and "hero" and "oh thanks for finishing Red Dead Redemption after months of crunch time, here's your pink slip [2]." It has to mature, at some point, I just don't know when that will be.

[1] http://www.zenetproject.com [2] http://www.next-gen.biz/news/rockstar-san-diego-confirms-lay...

I get your point, but I think that's really not what Michael Abrash was meaning here when referring to "heroes". His point is simply to show how hard it can be to finish a product, no matter how well or ill managed the team is.

While yes, Abrash's point was that you are a hero because you finished your project on-time, patio11 has a great point.

There have been many a horror story of long long LONG hours posted on here ranging from RockStar (lost the link) and many others because of a looming deadline with no working benefits for those extra hours.

Exciting as it was, we hit the same rough patches toward the end as any other software project. I am quite serious when I say that a month before shipping, we were sick to death of working on Quake.

Finishing projects should not require this and anyone who tells you different is your enemy. I mean that.

> Finishing projects should not require this and anyone who tells you different is your enemy. I mean that.

Enemy? How melodramatic.

Finishing complex multi-year retail projects is never easy. Someone will have to fix those nasty, hard to reproduce crash bugs, find ways to free up another few megs of memory, smooth out those frame rate spikes, etc. It doesn't matter how many politically correct agile practices you follow; you will always be left with critical last minute tasks that are anything but fun and relaxing to complete. You can choose to work on the kinds of projects where this doesn't crop up as much, but games with blockbuster aspirations were never that kind of project.

My theory is that un-released software is like physical inventory. Keeping inventory is very expensive because the money spent could have been earning interest elsewhere and product sitting in the warehouse becomes gradually obsolete. As you accumulate a great deal of software inventory towards the end of a big project the cost of keeping the inventory, and therefore the pressure to complete the project, naturally increases. Shipping small increments frequently and not keeping much software inventory is the most efficient process if the product design/market allows. (I'm not saying this is always possible or would have been appropriate in this case, just making a comparison.)

This article resonated with me and it has nothing to do with poor management or exploitation or enemies: for some of us starting projects (personal or otherwise) is always a lot easier and more fun than finishing them. Finishing projects can just be hard. And this isn't just within the realm of programming, although there I find it especially true.

I don't think that's true. Shipping is really hard - basically the end result of execution. A perfectly run project can still suck to work on at the end. The problem is that for most people the fun part of working on a project is the discovery phase when you're building something new. The last 20% of squashing bugs and going from "the code is working" to the "code is shippable" is generally not a good time.

I speak from a point of ignorance, I've never worked in the field. That said, if a project requires months of unplanned crunch time, it wasn't perfectly run. And I don't think anyone would say that crunch isn't commonplace.

(At least, I think patio11 is referring to the culture of crunch time. Correct me if I'm wrong!) You probably know that, but, if not, (or for those who don't) here's some reading:

http://en.wikipedia.org/wiki/EA_Spouse

http://arstechnica.com/gaming/news/2011/05/the-death-march-t...

I've read those articles when they were written, and have even read the book Death March. Those situation are obviously not how things should be.

Patio11 responded to Abrash saying near the end of the project there were rough patches and that eventually they all were tired of working on it. No matter how well run a project might be there will always be rough patches and you'll likely be sick of working on it at the end. Quake is a great game, but imagine running the same level or even worse, same portion of a level all day long looking for obscure bugs. Even if you planned for this time, doing this for weeks is going to suck at some point. It's just the nature of working on a long term project and taking it from working to shippable.

In the past I would agree with you but nowadays I've come to the opposite conclusion.

Shipping crappy, trivial, mundane, or even good software shouldn't require this.

But shipping something that is timeless is absolutely mentally, physically, and emotionally draining.

I think around eleven men died building the Sydney Harbor Bridge. I'm sure many men will die in the next few decades as we start to commercially explore space. Few programmers will die shipping a breakthrough product, but to say it won't require a heroic effort I think is to say it's not a breakthrough product.

The production of Apocalypse Now was by any account HELL. Far worse than any game development example I can think of. (Well, DNF jokes spring to mind, but at least they essentially built multiple games there. Apocalypse Now wasn't released until two years after it finished shooting.)

It didn't chase Francis Ford Coppola away from film. It was an isolated example of how bad things can get, a living worst-case-scenario. It wasn't "Tuesday at the office."

Are people dying during game production? No. But they're not in experimental spacecraft, and they're not balancing on I-beams hundreds of feet in the air, either. Given the sleep deprivation you hear about in game production, if they were in either of those scenarios? Yeah. Games would probably take more than eleven lives a year.

/edit: Also, do you consider the eleven men who died during the Sydney Harbor Bridge heroes? Or guys that got stuck with shitty, dangerous jobs in search of a paycheck, for whose deaths I feel more pity for, than respect? I'm more with the latter. Astronauts, well, that I can kinda get as a "hero". The gang that made Guitar Hero: Van Halen? Not on that level.

> Also, do you consider the eleven men who died during the Sydney Harbor Bridge heroes?

Kind of off topic but to answer your question:

If you define hero as someone who puts forth a heroic effort, then sure I would. Does them dying make them heroes? No. Maybe someone was hungover and slipped. But the type of work they were doing I respect.

If you defined hero as a role model, I have no idea of course (didn't know any of them).

"Few programmers will die shipping a breakthrough product,"

Not right away, but abusing your health takes its toll.

Extremely difficult technologies can be developed safely and correctly. The key example is the US Nuclear Naval Reactor Program.

No accidents. No injuries. No deaths. Thousands of years of reactor-time.

Okay, this is a bit sad. How would you respond to your employer if he used this "heroic talk"?

I couldn't disagree more. Doing the grungy work of getting a big project ship-ready doesn't automatically mean there's a death march underway. It means you're a team that's disciplined enough, grown-up enough, to focus on the good of the project, as opposed to what's fun.

There's no reason that kind of work can't also be rewarding. Running a marathon is hard work, not pure fun, yet people find the endeavor rewarding.

They may be your enemy in the sense of an employer / worker bargaining situation, but they may also be right.

Even in a well-run project, there are many factors that are outside the control or reasonable knowledge of the project workers (the unknown unknowns).

As far as I could see the article had nothing to do with exploiting programmers with visions of glory, but rather urging developers to finish what they start - a surprisingly difficult goal in our field.

This is true, but there's the question of what you do once you're in the situation.

I've been on fucked projects and projects that shipped industry-leading products. Success doesn't seem to correlate with a well-planned project. Instead it seems to correlate with how bloody-minded the participants are about shipping and moving forward from where they are.

This annoys me, as I used to be quite the process/method evangelist, and of course cynical project managers exploit this.

You don't get to be Carmack or Abrash by working 8-5. They didn't have a slave driver boss standing there making them work, they did it because its what they love doing.

A corollary of your point is that you don't get to be them by working 7-7 six days a week. You actually have to love what you're doing, which can be hard for the best of us after a six month death march.

I think you're thinking in the wrong context. You should be applying this to your own startup or weekend project.

While I have worked a heroics shop and learned to not do that again, I got a completely different read of that. It is a PG real artist ship speach not a to the death, or for queen and country speach.

Reply on news.ycombinator.com