Disney rendered its new animated film on a 55,000-core supercomputer

And more transistors doesn't mean more performance. Why do you think modern CPUs have 6 or 8 cores on? Because we can fit more transistors on a die but CANNOT make individual cores go any faster.

You can massively parallelize rendering a movie in advance because you can do each frame on its own CPU. Rendering in real-time is much less easy to extract this kind of parallelism from, particularly if you have hard real time constraints.

There's a great deal of iteration (look development, lighting) past animation approval.

Yeah, previs (pre-visualisation) is generally always done these days, as it lets the director preview what things "look" like, but that only has very basic geometry, low res textures (if any) and no proper cloth/skinning deformation animation, and no proper shaders / lighting. So it's basically using a game engine without even game engine decent lighting / shading for the most part.

So while it's great for placing objects in the scene and getting the camera position correct, that's about it. There has been a move to using progressive raytracing more and more over the past year or so, but it's still very early days with that.

On top of that, as previs is almost always done early (before most asset generation - modelling, texturing, maybe animation too) is done, it doesn't always give you all the info you need - i.e. it's often the case with mechanical hero objects in the scene (i.e. aircraft, robots, weapons, etc) that the basic geometry used for previs isn't good enough, and later on in real lookdev/lighting (or sometimes even at animation stage) serious issues are found - i.e. the previs has a robot moving through a street, but when animation actually start trying to rig the robot to get it moving realistically, they find for the arms to swing, it can't actually fit in the street.

So there's a huge amount of iteration that works up-and-down the pipeline, sometimes causing lots of different parts of it to re-do work.

Generally it goes: Previs -> Modelling / Texturing (linked, as you need UVs) -> Layout -> Animation - > Lookdev / Lighting -> Compositing.

Any non-trivial change before Lookdev / Lighting will trigger new renders having to be done for some layers / scenes.

The above poster is right. It's not just some iteration, there is tremendous iteration. The quality that you see comes from iterating until there are no more imperfections and that takes a lot of iterations and a lot of renders. A typical lighter may have 4 shots they are working on at one time, with one finishing about once a week.

So basically the impression from the DVDs is a simplistic fantasy. While the narration says 'it's such a wonderful place' the people working there until 9 every night are thinking 'why is everything broken'.

My bad. It has been a dogma for decade(s) now that production quality feature is from 0.5 to 2 hrs per frame for final though, so that was my reaction without reading correctly. You have to factor in numerous passes and production previews which add up. source: am rendering daily.

TV work, on the other hand, can't deal with that kind of time burden, so it's shifting to more novel solution which lately involves near real-time rendering (see element3d for example).

No, layering just means generating multiple renders for a given shot, which can then be combined to create the final image using 2d compositing software such as Nuke. Full 3d renders are always going to be much slower than just doing image manipulations in 2d, so it is much more efficient to iterate by making adjustments in 2d than by re-rendering everything from scratch every time you need to make a change. Additionally if just one element in a shot changes, say a character's animation changes, it much more efficient to just render the layer for that element again than to re-render ALL the geometry for the entire shot. Each layer may still be rendered with full global illumination and whatever other fancy lighting is needed.. the layering just makes it easier to iterate in individual pieces of a shot.

In feature animation things are changing and being adjusted constantly. Each shot is worked on by many people across multiple disciplines, all of whom do many, many iterations to hone in in the final shot. So the entire production pipeline is designed to make handling changes as efficient and easy as possible. Look at the amount of processing resources disney used for this movie.. I can guarantee you they are cutting all the corners they can and maximizing efficiency wherever possible. Doing a single monolithic render every time something changed would be extremely inefficient.

That said, there are a few studios that do produce complete final frames in-render. Usually places that write their own renderers in-house and therefore have a sort of macho academic attachment to showing off how much their renderer can achieve out of the box. Blue Sky is this way because their whole pipeline is designed around their cg studio renderer. Anecdotally I have heard that Pixar did everything in renderman for a long time but more recently they have started using more of a compositing workflow since it's so much more efficient. So it's possible that disney may be this way now since they have this fancy new hyperion renderer they wrote, but I certainly wouldn't bet on final frames all being done in-render.

In seti@home, they are all being controlled from a single location to work on a single task, while the entire internet consists of independent machines operated independently. That's what allows you to talk about seti@home as a single supercomputer, even though it's somewhat tenuous as the problem is so embarrassingly parallel, it would never compete on any actual supercomputer benchmarks which require low-latency communication between nodes.

The first cg company to use 10GbE? I find that hard to believe. I develop storage boxes for video production, and 10GbE is pretty much standard these days, you only go with gigabit if you're being really cheap. We are now starting to sell 40GbE systems on the high end, as well as bonded 10GbE for those that need a bit better than 10 but don't want to shell out for all of the really expensive 40G gear.

Disney probably would have had to mortgage the castle to afford rendering the movie on AWS.

> Nobody uses EC2 for this type of workload.

I don't understand how people make assumptions like this, unless they work at AWS at see the customer account info and real world workload data.

Houdini Cloud specifically uses AWS.

http://www.sidefx.com/index.php?option=com_cloud&Itemid=213&...

Yes and no. Primary rays cache, but secondary trash. That's why CPUs are still king for that type of rendering.

Forgive the off-topic post, but can I email you privately? I would like to license your nickname for use in a legitimate business.

They don't do final renders immediately. They render previews with expensive features like global illumination turned off.

Rendering a single frame across multiple machines sounds wasteful. They would have to load the exact same textures and models for a single frame across all of them. When batch rendering, it would be more efficient to do that work just once per frame.

I can't go into any specifics, but the general approach that most places use these days is to assign complete frames to each node on the renderfarm. The nodes are multi-core and threads divvy up the work of rendering the image in tiles. That's not to say that production renderers can't do multi-process renders too, but you don't see it nearly as often as multi-threaded renders. Remember that it can often take several minutes just to read scenes in; even if you split up an image into individual pixels and gave each node just a single pixel to do there'd still be a limit to how quickly you could get a frame back. So if you parallelize across your cluster on frames, then yes, there's usually several hours latency for a frame sequence, but you also get much better throughput.

By the way, if you haven't heard of Blinn's Law [0], you may find it interesting.

[0] http://blog.boxxtech.com/2013/07/15/blinns-law-and-the-parad...

This is not true. Frames are queued to computers with free cores and memory. Splitting up frames across multiple computers is a brute force option that is rarely used. When you get to that point on show it is because render times are huge and often because renders have failed and need to be forced through during the day to make a deadline. It is a telltale sign of disaster show.

On an individual box frames are split up into tiles and the tiles are rendered on individual cpu cores.

I don't know what you mean by "a few" frames per day, but it sounds like you have a misconception about how it works. Typically many, many thousands of frames are rendered every day, and most of those are rendered multiple times as artists iterate.

The unit of parallelization for renders is always the frame. On a machine with multiple cores each core will render tiles in parallel, but across machines jobs are split up by frame. This is both because it's the simplest type of distribution for people to understand (machine goes bad? You lose one frame, not arbitrary pixels in an image). But also because it's most efficient for a single machine to read all the data for a single frame instead of multiple machines requesting the same data repeatedly across the network. I think people tend to underestimate just how massive the geometry and scene description files are for a typical feature and how much of the work involves managing the storage and network efficiency.

It's done on frames because that's how renderers work. You build a description of a single image and the renderer renders it. The renderer will generate threads which will all render a 'bucket', which is just a square block of pixels in the image.

It's possible to parallelise this across several processes on different machines, but this is obviously less efficient because you'll have to do all the render startup tasks (reading resource files, building acceleration structures) multiple times.