Comment by kevingadd | Hacker News Reader

Comment by kevingadd | Hacker News Reader

A 15% framerate increase from 3 weeks of work isn't particularly impressive. It would be more meaningful if the measurement was in terms of frametime (i.e. elapsed time per frame) since framerate is not linear.

From looking at the patch (nice of them to provide it), it seems like most of the changes are putting mutexes around things to guard against simultaneous access and then parallelizing some loops. It seems to me like anyone with basic knowledge of the game's codebase could have made these changes without any support from a tool like the one they're trying to sell, and probably done it in far less than 3 weeks. If your goal is to run a piece of code in parallel, you don't need a program to walk through that code and identify all the shared state it manipulates (and then go through and make sure that state has a mutex guard). If the software they're trying to sell did that for you, now that would be nice. It seems unlikely that you could automatically modify C++ source code without a lot of constraints, though.

Software to aid authoring parallel C++, or detect places where you can safely parallelize existing code, would certainly be valuable but this is a pretty poor demonstration of any actual advantages offered by their software.

EDIT: From watching the included demo video, the game is running well below 60fps which suggests that they're running it on a low end graphics card (or generally low-clocked machine) to improve the gains from their optimizations. I don't think their results are necessarily real-world as a result of this - someone playing Doom 3 on an actual gaming rig would probably not see a 15% framerate increase because their framerate would already be well above 60fps (framerate is nonlinear, as I mentioned above).

This is not just middlebrow dismissal, it's outright wrong. In the AAA world we would love to spend only 3 weeks of an engineer's time to get a 15% speedup. Seriously, that is a great deal, it's like, where do I sign up?

However, it becomes substantially less impressive when you notice that you're using 2x or 4x the amount of processor hardware (2 or 4 cores) and only getting a 15% speedup. In a by-hand implementation that would be very disappointing.

If it were fully automated that would still be pretty valuable, but it appears that this isn't. So it seems to be of questionable utility.

When were you in the AAA world, Jonathan?

I did a lot of consulting / contract work at AAA companies between 2000-2005. "Parachute in and make the E3 demo work / implement some tough feature / etc". Sometimes it was less well-defined than that.

Woah you're the guy that did Braid - glad to see you on HN :D

Love to hear more about your 2000-2005 paratrooper work at the AAA shops - have you discussed that at all anywhere?

I've always been interested in seeing how the really tough stuff gets done at AAA shops.

The same way it gets done elsewhere - you find a person who's an expert at that kind of stuff and tell them to get it done :)

The parachute-in often happens for teams that don't have a whole staff of low-level engineers around. Either small studios using an off-the-shelf engine (that can still be customized), or studios in a large conglomerate. (EA e.g. used to be quite happy to ship experts around. At least back in my day.)

Worse, they didn't actually get a 15% speedup. They reduced the frame time from 16.7ms (around 60FPS) to 15.3ms (around 65 FPS), which is only a speedup of something like 7%.

They didn't seem to mention how much they're actually utilizing the remaining cores either. From the look of it, they've basically parallelized a couple of loops in the renderer. That strikes me as being the wrong way to go about this, especially considering that modern AAA games apparently break everything down into tasks, and run the separate tasks in parallel (idTech5 apparently does this, for example).

p21 in the white paper suggests the speedup is ~2x, for the parts that were actually parallelized. That's more interesting (well, to me) than the rather limited increase in overall frame rate.

You may be right that they are doing the wrong thing, but I think the fact it's doable at all is reasonable evidence that their tool is useful. I'm pretty impressed that somebody who's totally unfamiliar with the code is able to jump in and start parallelizing stuff - particularly if they've never worked on a game before, and so might not have any real idea where would be a good place to start, or how things are likely to work.

Maybe my standards are too low.

(Those screwy HUD textures might be evidence that they managed to get it completely wrong, of course ;) - or maybe it's just something simple.)

Good point. Their tool is obviously useful, even though something like this might not be the best use case for it.

The HUD thing - it looks like they screwed up the post-processing effects somehow. As far as I remember, the original game did apply post-processing effects to the HUD. Probably something to do with OpenGL not getting a decent way to do render-to-texture until 2005, so they'd have had to use the previous frame's render buffer. It wasn't that noticeable in the original though.

The exercise was more about evaluating what the tool could bring for already-optimized code than about comparing the results against what an expert programmer could achieve manually.

It is true that the application domain of the code chosen for the blog post might be a bit unfortunate in order to make a realistic case. Game engine programmers are usually very good at low level optimizations. Pareon can be used by non-experts though and additionally assist experts in during the task of finding and applying optimizations and paralellization oportunities.

Quoting the blog update [1]:

"Our Pareon tool is not aimed to replace parallelization experts. We want the tool to be useful for any programmer who is looking for paralellization opportunities in either his own code or in code that he is unfamiliar with. A domain expert (game programmer) can probably save some time because of his in-depth knowledge."

Additionally:

"The three additional cores on my four-core machine are only used during part of a rendering cycle. They are sleeping the rest of the time. If I would parallelize other pieces of code in the cycle as well, it would increase utilization and I would expect the frame rate to improve even more."

[1] http://www.vectorfabrics.com/blog/item/accelerating_the_idte...

Came for the dismissive hand-waving that I fully expected to be the top comment and I wasn't disappointed.

What's the improvement factor where you'd be impressed? 50%? 60%? It's remarkable in my opinion how little work it took to improve upon an already very-optimized engine, in the constraints that they set for the developer, by a developer who's never worked with game engines in his career.

Edit: No longer the top comment; it was when I commented.

Framerate isn't a comparative measurement. You have to measure elapsed time.

Instead of spending 3 weeks wrapping data structures in mutexes for a 15% framerate increase, you could probably get an equivalent speedup by reducing shader detail, texture size, or any number of other options. This kind of measurement is not meaningful without additional information: Is this improved performance for low-spec GPUs? High-spec CPUs? universal performance improvement for all users? Low end machines typically do not have lots of cores; will they actually benefit tremendously from parallelism? As I said before, a high-spec machine already runs Doom 3 fine, so I question whether the gains here would actually be significant at all.

EDIT: Also, gains of this size can easily be produced by changes as simple as turning on Profile-Guided Optimization in your compiler. We'd also need to know what settings they used to compile the game to know whether the results are actually realistic.

> Instead of spending 3 weeks wrapping data structures in mutexes for a 15% framerate increase, you could probably get an equivalent speedup by reducing shader detail, texture size, or any number of other options.

I see why we differ in opinion here: you feel that regressing backwards and sacrificing visual quality is a better approach than parallelizing the same exact engine with the same quality. I don't think we're going to see eye-to-eye on this one.

PC game development is about delivering a good experience for users on a wide range of hardware configurations. A given game is not going to automatically run perfect on every hardware configuration.

When considering performance issues for a particular customer, you have to weigh the cost of improvement. 3 weeks of an engineer's time for 15% speedup (even if it were in elapsed frame time, which it's not) is NOT an easy decision. Those 3 weeks could be spent doing far more valuable things, like improving the tools used by the rest of the development team, fixing crash bugs, or working on downloadable content that will bring in more money for the studio. They could also be spent making architectural changes that provide much larger performance wins through design improvements that require actual knowledge about the design of the application and the significance of the choices made.

When you're doing performance optimization in game development, you go for the biggest, cheapest wins first, based on actual measurements and an understanding of what's wrong. Given that the optimized version of the game is not running at 60fps (and the demo video contains obvious rendering glitches) I don't think it's unfair to question whether CPU parallelism was the most obvious bottleneck here.

Lastly, dropping texture size or shader detail when running on a low spec machine is not a regression. If the machine is not capable of running at max rendering detail due to GPU constraints, no degree of CPU parallelism will overcome this.

All of what you say is wise, but the engineer doesn't work for the studio and isn't on Doom 3's release timeline. He made the game's frame rate 15% faster of his own volition simply because he could. So it's wrapped in a sales pitch, oh well.

"I see something I can improve, but I had better not improve it; were I to work at id, my time would probably be more valuably spent on other things."

I wish you'd just back off and laud the improvement rather than fitting it into your model of game development and continuing to double down on a snarky, dismissive comment that basically shits on somebody else's work. I certainly appreciate that you've worked in professional game development, but come on, someone did something cool. I'm sorry that it isn't cool enough for you or doesn't use the measurements you prefer.

Attitudes like yours hurt our industry as a whole.

This article is clearly trying to sell licenses for a commercial product based on unsupported, possibly deceptive claims. I don't laud that. If the goal here is to prove the value of this analysis software, they should be proving it in terms of the value it will produce for an actual company developing software - if they're proving value for a game studio, it should probably be in the context I provided above. For other kinds of developers, maybe they have infinite time and their constrained resource is engineer knowledge - in that case this software could be valuable. But you don't demonstrate that using video games.

Hacker News isn't about selling things to managers who make purchasing decisions by lying to them, last time I checked.

Oh, now they're lying? Care to back that one up? That's a steep accusation that you should probably reconsider attaching to your name.

They wrote a tool to find things to parallelize. They parallelize them, the code gets faster. I realize that it isn't the faster you would prefer, but they picked an accessible target that everybody could understand. It isn't as cool if they say "we made Excel render graphs 15% faster" or "we made MP3 transcoding 15% faster". They just made a code base faster using their tool, that's all.

If they had made a video editor faster and you'd worked on video editors all your life, I'm sure you'd be here shitting on the achievement as well. They're not a game development shop. They picked a game to screw around with. They made it quicker. It just happens to be your pet area, so you absolutely cannot stand that someone figured out how to do something in your little world that you wouldn't think of.

I'm impressed you've upgraded to outright calling them liars now rather than just admit, maybe, you might be wrong.

> He made the game's frame rate 15% faster of his own volition simply because he could.

This isn't some lone engineer's noble cause. This is a sales product demonstration.

> Attitudes like yours hurt our industry as a whole.

No it doesn't. People SHOULD be critical of demonstrations like these because they are rarely representative of real world behaviour.

People who have never used software should be critical of "here's what our software can do with inexperienced hands, maybe you'll do better"? I'd much rather default to giving benefit of the doubt than assuming everything sucks before I've even touched it. Reminded of people who say they don't like exotic food but have never eaten it.

I think the lowest opinion on the totem pole is one formed by assumptions (based upon personal experience with an area) used to reflect upon something from the hip. "I'm pretty versed in game development, and what these people are doing is stupid based on my world view."

Wow. You really invoked some hates. All your long and thoughtful comments have been voted down. I would say you have hit a nerve.

I, like pg and hopefully other Hacker News readers, am pretty sick of every story about something cool being immediately derailed by a know-it-all in an industry dismissing the innovation as "not cool enough". pg calls it middlebrow dismissal[1]. Rather than being treated as an imperfect launching-off point for further innovation, everything has to be world-changing perfect to have a reasonably sane comments section.

I'm actually glad this thread got pushed down, so maybe some actually informative discussion can rise up rather than an analysis of how minutely the people behind this screwed up.

[1]: http://news.ycombinator.com/item?id=4693920

Well, there are a lot of Apple fans here, and a core part of the Apple aesthetic is "if it isn't perfect and doesn't have that 'wow' factor, better to leave it out".

The only way you know if your product is imperfect is by other people constructively criticising it. But instead you would rather all of us stand around in a circle cheering them on about how wonderful it is and politely ignore any problems.

Sounds like a guaranteed way to fail.

Dismissal is not constructive criticism. There's a sizable difference between "meh, nothing to see here" (which is the original comment in this thread) and "here's how this could be better".

Dismissal is absolutely constructive criticism if you're looking at it from a business perspective.

If the product does not provide a decent ROI then you are right to dismiss it.

Ruling out a product and dismissing it based on projected ROI is not criticism. That would be called "making a business decision," not providing constructive feedback.

Another negative trend that's clearly happened in this thread is drive-by voting: all of your interlocutor's posts have negative points, whereas the only posts in the thread that deserved any downvotes were the initial post (the only legitimate instance of the poorly-named 'middlebrow dismissal' that I can see) and the one where you put words in his mouth at the end ("Oh, so they're liars now?"). Postulate: very few people read anything but the first post and its reply, and upon doing so most upvoted all of your posts in the thread and downvoted all of your interlocutor's. Intelligent discussion indeed.

I didn't put words in his mouth, he wrote that they were lying to managers and possibly being deceptive. You also misquoted me, which is ironic given your criticism.

Your postulation is unwarranted, by the way, since my posts have been going up and down based upon their depth in the thread. The top comments have been upvoted more than the lower ones, and the lower ones have actually been muted somewhat. I would actually suggest that people are reading the thread and voting appropriately, and probably aren't making it deepest into the thread.

Your sweeping characterization of mindless Hacker News voters acting by whim is disconcerting, to say the least. That we're so fixated on the voting here when karma is absolutely worthless is equally annoying. I am, however, intrigued by the shift from very light gray back to legible that has taken place on his comments since the beginning of this meta discussion (starting with ww520's remark); there might be something to what you're saying, which is that people are acting based upon what others are saying -- that's troubling.

When I see dismissals I look for a particular kind of fallacy, which this is an example of: Reasoning backwards to support the opinion. "This tool is unknown to me and their claimed improvements are dramatic" - "I would not trust an unknown tool and I do not trust their marketing efforts" - "Here's some mix of facts and expert's assumptions that supports the idea that you can't trust it."

When you're reasoning backwards you're on thin ice because it's easier to let a logical fallacy through as you start assembling your "facts and assumptions" in the rush to make your point known. You can "win the battle" (by being quick) but "lose the war" (by being wrong), and when I catch myself doing it I have to either cancel the post or put extra effort into it to make sure I have a valid argument(and often, after doing enough research, I don't, or I have gone too far outside my domain to know for sure).

Forward reasoning usually results in very straightforward critiques like "I used this but it was not appropriate for these situations..." or "it completely failed in this case..." or "it turned out to be unnecessary for the project I was on." Kevin's posts(both the initial one and follow-ups) get muddled because he has to weave together several minor points; as each successive rebuttal comes forward, so does his target, so that the final opinion remains the same, even though by the end he's reduced to "they're lying."

(A good example of a game developer who has put some serious effort and notetaking into finding ROI on a new tool is John Carmack and his forays into static analysis.)

Reply on news.ycombinator.com