An Analysis of 155 Postmortems from Game Development [pdf]
research.microsoft.com
research.microsoft.com
When things don't work out according to the schedule, there are generally two reactions :
1. We didn't estimate well enough, we need to estimate better next time.
2. Maybe this estimation thing is impossible to get right, let's not put too much trust in being able to estimate accurately, but rather find ways of working that don't require estimating all work up front.
I've seen that 1 gradually becomes 2 with experience (the experience of getting the estimates wrong every time).
anything beyond that scale becomes the educated guessing game. no amount of analysis can flesh out every blocker that will be encountered. the deeper you go analyzing each task, the more the work just becomes doing that task.
I thought the same thing as you, but a month ago someone explained it to me in a way that makes a lot of sense https://news.ycombinator.com/item?id=11641316
Actually a lot of the comments about this PDF can be summed up with "practice Agile development"
I think that toolchains should have better support for estimated vs used time and then learning from that. You could classify tasks by field (DB, frontend, etc) and then calculate correction values based on past failures. Learning from wrong estimates seems to be even harder than estimating, because it hurts to admit that you were wrong, so some automated, blame-less version of it (per sprint or so) might be a good start.
Of course budgets and deadlines has to be set. Agile is not a way to ban those, it's a way to allow them, even if forecasting is impossible. It's about saying, well at that date I can promise to have kept under budget, and will have something that more or less reaches the goal of the budget. It will not, however, be what anyone is imagining right now, let's have fun discovering what it'll be.
Isn't agile the #2 option quoted above? Agile in game development means being able to estimate how long it'll take to add fire damage to a weapon. It's not a means of being able to budget out 3 years worth of development, let alone hit a shipping date that far out.
In a perfect world, it works as other comments point out: you get better at it, and everything is wonderful.
In real life, that estimate you gave last week becomes your duty. Unless you pinpointed it, you're out of schedule. If you're late, you look bad. If you're early, you get more work next time.
And there is no good way out, you can do it poorly but in time (which will only compound time as your poor solutions will introduce more work). You can do it late but good (but, well, late). You can give stupidly high estimates and then make your own secret schedule (and get found out eventually).
Agile is Fordism.
My advice: do not estimate timings. Have a list of things TODO, and have someone monitor expectations.
Given that, a precise understanding of the time involved to accomplish a task requires knowledge of the time it takes to come up with the correct instructions + the time it takes to give the correct instructions to the computer. The time cost to give instructions to a computer is effectively linear in the size of those instructions (its whatever my typing speed is times that size), but the time cost to understand what those instructions need to be is effectively unknowable unless I've already got those instructions. It's sort of like the halting problem for people... if I knew how long it would take to come up with a reasonably precise version of them, you'd have a version of those instructions that could execute in some language (say some imaginary pseudocode language), or I'd have determined that the task is impossible.
I can keep a heuristic understanding of how long a task will take based on previous tasks I've completed (i.e. this task sounds similar to this previous one, which took me this long. I'll use that for my estimate), but there will always be some set of tasks for which I'm wildly wrong unless I precisely examine what's involved.
If it's better to estimate in a fashion such that I deliver everything I promise (and that we can have a planning session that doesn't last the entire sprint), I can pad my estimate by a large factor. In general, its easier to meet the criteria for this situation, but there will still be tasks for which my estimate can be wildly off. This can also lead to unhappy product owners who assume I'm trying to rig planning to excuse less work.
How do you come out of this exercise unscathed and with valuable information?
But unfortunately this title is some kind of cruel joke:
http://www.phobe.com/sfi/ornithology.html
http://www.gamasutra.com/view/feature/130925/schandenfreudia...
Found it: Pacemaker - putting the heart back into the healthcare industry. https://youtu.be/jG9BgjnpGGY?t=940
> Finally, based on our analysis of the data we collected, we make a few recommendations to game developers. First, be sure to practice good risk management techniques. This will help avoid some of the adverse e ects of obstacles that you may encounter during development. Second, prescribe to an iterative development process, and utilize prototypes as a method of proving features and concepts before commit- ting them to your design. Third, don't be overly ambitious in your design. Be reasonable, and take into account your schedule and budget before adding something to your de- sign. Building o of that, don't be overly optimistic with your scheduling. If you make an estimate that initially feels optimistic to you, don't give that estimate to your stake- holders. Revisit and reassess your design to form a better estimation.
It's amazing how common-sensical this all his. But then again, we could all use a little more common sense sometimes.
As Charlie Munger would put it: "It is remarkable how much long-term advantage people like [Warren Buffett and myself] have gotten by trying to be consistently not stupid, instead of trying to be very intelligent."
Unfortunately our system has created perverse incentives: What manager or entrepreneur can afford to take a step back and say "what I'm proposing may not work, let's stop the project"? We have chosen to work under a pressure as if our lives are at stake (and they are - not our bodies, but our livelihood). The guy selling worthless shit, getting people to buy stuff they don't need, that may even be bad for them (or for those producing them) does not have problems. The same guy voluntarily giving up his job where he would have to do things he thinks are actually bad and/or useless is seen as a "parasite" on society - he's not working and we still have to feed him???
You bet all my estimates show the project is worthwhile if I don't have an (equally good) alternative.
In the early years of the PS3, game projects were failing because the Cell architecture (a small PowerPC CPU plus six auxiliary Cell processors with tiny memory) required completely new game engine architectures. That was finally figured out, but development on the PS3 seemed to run about two years late, which set Sony back.
Now there are mature game engines, mature physics engines, the graphics pipeline is in good shape, and the hardware is powerful enough. Problems in game development now are more about the game than the underlying technology.
(Pre-1990s, game development was a cram job, trying to get stuff done on limited hardware.)
The ports you reference behave like game systems that used to occupy 100% or almost 100% of the resource capacity of a dedicated system failing to execute correctly when they're running in a non-dedicated environment.
They also had the advantage of targeting an exact set of hardware specifications in other ways and didn't have to scale down, up, or sideways (other ways of dividing labor) to the way that other platforms work.
In short, they weren't a game designed to scale to a theoretical platform that then ran on the platforms, they were fit precisely to a single platform and then weren't designed to be portable at all.
I didn't quite understand the correlation between development for multiple platforms and quality of art. May be those who want to release for multiple platforms in general care about quality more?
However I think the authors took a bit too much liberty with the conclusion (i.e. "may have better game art"). It sounds more like the designers of multiplatform games may just have a higher opinion on the art in the game, due to bias.
Or at least, that was the case last time I read the gaming press about 10 years ago. I don't know if it's still an issue these days.
Only now consoles seem to be speeding up, possibly because of increasing competition like Steam Machines.
Another problem were interfaces. Too many developers cut corners and didn't develop separate interfaces for controllers and keyboard & mouse. This resulted in very bad UIs.
> Takeaway: In order to avoid conflicts during the development process, teams need to have proper management.
This seems like the exact opposite of Naughty Dog's advice
http://www.latimes.com/business/technology/la-fi-tn-naughty-...
Takeaway: Game developers should create a well-defined concept before beginning development as opposed to an ad hoc method of game design.
This would exclude Dota, for example. Gamedev is a world of exceptions.
In general, it's best to playtest every idea you come up with and cut whatever isn't fun.
Also, the advent of engines like Unity and UE blur what "modding" means. The WC3 engine was flexible enough to be made into Dota, which was then implemented on the Source Engine as Dota 2. But why was the original Dota a mod, whereas Dota 2 isn't? It's hard to come up with a definitive answer to that.
Additionally, the originally Dota started as a mod, was then published as a game, and Dota 2 would then not count as a mod. Its not hard to come up with a definitive answer to that. A game is a mod if it doesn't stand alone, if it requires _modifying_ the structure/code of another game.
It's better to say that mods are games and to treat them the same than to try to differentiate them. The same design process that yields a successful mod will yield a successful game.
But the second half is completely false. Rust has nothing whatsoever to do with DayZ, except that it's vaguely in the same genre.
That's different from the typical lifecycle of a commercial game, which has a deeper level of marketing work to do before it can even consolidate a playerbase. There isn't a "Unity modding community", per se. There is a community of Unity developers, and players of individual games made in Unity, who have no broader associations. A game transitioning from the mod sphere of an existing game to an independently produced commercial title still has to overcome this gap, and most of them don't make it over the line.
The advice is sensible, nevertheless. If you can extend the schedule to focus on the core elements of the design as a "R&D" process where the majority must be thrown away, and only scale it up towards a shipping product as the concept proves itself, you minimize the risk of the budget being wasted on a flawed concept. Modding scenes, game jams, and micro-budget productions all have the benefit of weeding out most of the really early, risky design experiments, without wasting enough of people's time or money to care.
Even when conceiving the marketing for a larger production, the same advice works. The "trial balloon" or "landing page" method, etc. You still do design thinking when you market, but it's design on the topics of "how do we build a funnel" or "how do we make this a franchise." Still very easy to spend a lot of money on making a splash without getting the blueprint right.
Playtesting isn't useless. It's how fun games are made.
Perhaps people are disagreeing with this because they haven't tried to make a game. A curious phenomenon.
"Listen to your users" is not advice, it's the end goal. What's the most effective way to listen to your users? Professional playtesting? Open alpha/beta? Forums, IRC channel, a subreddit? Frequently scheduled AMA with the devs? Infrequently scheduled AMA with the devs?
If you want to watch someone doing this, pull up some videos of Notch making games. That's the process he follows constantly, from start to finish. He also has good taste, which is why his games turn out well.
If you don't have a nose for what's fun and what isn't, you have to take feedback from your users. If it's a multiplayer game, you should take feedback from the most competitive users, ideally from the people who are trying to form pro tournaments around your game. Casual players don't care what you do to the game, because it doesn't affect them much. But competitive players are your lifeblood, even if you don't know it yet. A competitive scene is what carries a game after its shelf-life has expired.
Playtesting is key but its far from being a simple step in itself and it never tells you exactly what is the right solution to fix things. Thats why making games is hard.
You can only really create a complete "fleshed out" design upfront if you're walking a very well trodden path.