Interstellar movie black hole implemented with Einstein's equations in C++
twitter.com
twitter.com
From what I recall, I don't think that it was running on that many machines at once. Mainly because it required the high memory nodes that were expensive. I think it was only running on ~10 possibly 50 machines concurrently. But I could be wrong.
What it did have was at least one dedicated fileserver though. Each of the file servers at the time were some dual proc dell 1u thing with as much ram as you could stuff in them at the time (384 gigs I think). They were attached by SAS to a single 60 drive 4u raid array. (Dell PowerVault MD3460 or something along those lines. They are rebadged by Dell and were the first practical hotswap enclosure that took normal 3.5" SAS drives, that didn't cost the earth)
The array was formatted into 4 raid6 groups, and LVM'd together on the server. it was then shared out by NFS over bonded 10gig links.
Anyway. That simulation totally fucked the disks in the array. By the time it finished (I think it was a 2 week run time) it had eaten something like 14 hard drives. Every time a new disk was inserted, another would start to fail. It was so close to fucking up the whole time.
I had thought that the simulation was a plugin for houdini, or one of the other fluid simulation engines we had kicking around, rather than a custom 40k C++ program.
It also mentions that ‘several hundred’ of their 1633 10-core, 156GB (weird number?) blade servers were used, but didn’t seem to go into details on data storage.
Is it possible you were working at the compositing phase, which would have been very heavy on random read and writes, resulting in more wear on the disks?
I was a systems engineer at the time. Compositing is actually really quite ideal linear streaming. You read the images serially and tend to write them back serially. which for spinny hard disks is about as optimal as possible. each frame tends to be between 20-150 megs (resolution, layers, bitdepth). but they are rarely read in a random IO way, you start at frame 1 and go to frame
there were 32 primary fileservers at the time, two were spare, and, including the nearlines (which were _n_ lustre machines and each had 4 raid arrays attached by SAS rather than one) we'd normally expected to replace twoish disks a week.
They ran particle sims all the time. Water, smoke, explosions are all staples of VFX. It was just something weird about this particular sim.
My understanding was that the actual simulation was causing the disks to die, rather than the render. A render can be restarted, a sim, less so. Well, this sim at least.
Either the renders were heavily paging memory to disk, or OP was working at the compositing phase.
Long story:
Forgive me if you already know this, I'm going to start with a toy example and then ramp up the scale.
Imagine that you are doing a "normal" simulation of something like a ball rolling down a ramp into a pot. That's fairly simple, but you need to know the position of the ball, the force vector and the location of any object near by that might collide with it.
It gets a bit harder when there are two balls, as they might also interact, so you need to store the state of two balls.
At a thousand balls, you start to need to think about scaling/threading/parralising (I mean you probably don't, you'd just use physX from nvidia and make it the library's problem, but bear with me) One way is to divide up the simulation area into a bunch of voxels and treat them as individual processing areas. When an object enters/exits an simulation area, you pass the object over with its force vector, and mass, and let that simulator deal with it.
Now, I'm not a physics engine person. The above may or may not happen as I describe, but the key thing is, there are sub simulation to allow you to run in parallel. You need to make sure that each processing voxel has completed it's processing for that time step before you can move on.
Once you have calcuated the position of each object, you can save that to disk and begin to calculate the next frame. Note, this isn't rendering, this is just working out the "pose" (position and rotation) of each active object.
The simulation of the black hole was effectively a massive particle simulation. I don't recall how many particles, but something like 14 billion sticks in my mind. It might have only been a billion. So they need to calculate and store the id, position, rotation, velocity and probably other details for every single particle. Even if everything is in a single float for a billion particles, its 28 gigabytes for position and rotation alone (xyz for position, quaternion for rotation) That's without any other state info like heat, force vectors, the weird quantum stuff or weight.
Once you have the position of all the particles, you then need to fire photons through them to work out what colour the pixel should be. That involves loading in the position of the particles, firing rays through them all and recording what it hits and why.
But why so much file IO? didn't the machines talk directly?
DNEG at the time was as close to being a "perfect" unix/linux shop as you could get. everything was ephemeral, even the file servers. All the nodes in the render farm could configure themselves from scratch (pretty much) from power on. You'd plug it in, switch it on and it would work out what it's hostname is, image themselves and join the rendering system autonomously. Most binaries that you used were stored on an NFS share somewhere. this meant that you had more or less complete control over the data flow. For example `rm` wasn't actually rm, it was a wrapper that moved files into a 'deleted' area, rather than binning them.
There were many layers of backup, all the way to tape.
Your home directory followed you everywhere, which meant so did your environment. This meant that if you wanted a specific version of a program, we had a wrapper that set that for you. So typing "maya" would spin up the right version of maya, use the correct plugins, and safe your files to the correct fileserver.
Everything was divided into shows. so you'd cd /shows/$showname/$department/$shotnumber/$version and you'd magically be on the right fileserver. This was done via the magic of symlinks.
So everything was files, and if you wanted to exchange large amounts of data, you'd use files.
If anyone is interested I'll talk about the system they used to coordinate all the rendering (no it wasn't k8s)
most certainly, please share. and thank you, this is very interesting.
So, the "farm" (the name given to all the machines that render everything) had 36k CPUs. I can't remember what the specs of the machines were but I think they were either 8 or ten core CPUs. Most of them were blade units, because that was the densest way to fit in that many CPUs into that sized space (The farm lived in the basement and consumed something like half a megawatt, I can't remember if that included aircon or not.)
Now, each machine on the farm was split into slots. From memory, the biggest slot was 8 cores, but you could request less.
THe farm ran "jobs" which were lots of commands strung together into a "direct acyclical graph" (DAG) for short. A node in the job could be as simple as "cd /show && mkdir dave/" or it could be a render. EAch stage in the job could have a dependency, either on a physical property, like amount of ram, or machine class (some CPUs were newer than others) a license to run renderman (a renderer from pixar) or some other expensive bit of software. It could also be dependent on a previous stage completing (so frame 44 can't render before frame 20 because it needs to reference something that frame 20 generates.)
All these commands are parcelled up into a single lump, using a "job description language" and sent to the scheduler.
Its the scheduler that works out where and when to place a command, on which machine. Now, the system that they used at the time was called alfred. The thing you need to know about alfred is that it's interface was written in something that looks like the athena widget set: http://appartager.free.fr/renderman/prman%2012.5/programming...
Alfred is old, as in, Single threaded, older than SSH old. The man page dates from 1995, and I suspect that its probably older still by a good 5 years.
However, despite being old, its still fast. It can dispatch jobs way quicker than k8s, even on an old shitty machine. But, we were pushing it a bit. I think we were sending something like 30k commands an hour through the thing. (ie, telling a machine to run a command, store the logs, capture the return code, pre-empt, reap, all that kinda jazz). We did have to run it on an overclocked workstation, as the main VM cluster wasn't quite fast enough in single threaded performance to keep up with demand.
WE had something like 800 artists in the building, all using the quaint athena interface.
There was a cgroups wrapper that was written to make sure that people couldn't take more ram than was allotted. We over subscribed CPU by something like 10-20%. If you went over your ram allocation, you'd get OOM'd. swapping ram between processes is expensive, swapping CPU is pretty much free (its not, but the penalty for running at 110% CPU is way less than paying the electricity for having more machines and undersubscribing.)
So why not K8s?
I'm sure some people do use it. But its not practical for batch processing like this for a number of reasons:
1) scaling past 500 nodes means that you loose a lot of network to message passing and state transfer
2) the scheduler isn't designed to have complex dependency trees (by default you can have a sidecar and thats about it really. You can create a service, but thats not really designed for ephemeral tasks)
3) the networking is batshit. (virtual networking is really not great for low latency high throughput stuff like NFS or some other file protocol)
What can you use?
If you're on AWS, Batch is good enough. Its not as fast, but it'll do. You'll need to write an interface to make complex job graphs though.
Azure has a batch interface as well.
https://www.opencue.io/ is what a lot of people use. And some people use https://renderman.pixar.com/tractor
Would be a better title
Chris Nolan doesn’t shoot digital. His more recent pictures (Interstellar included) are shot on a mixture of 35mm, 70mm and IMAX 70mm (!)
https://secureservercdn.net/198.71.233.183/gkz.aeb.myftpuplo...
The geodesic equations are not Einstein's equations.
The code for Interstellar only had to ray trace on a fixed analytical background spacetime (Kerr metric),
rather than solve partial differential equations for the metric itself.
[1] https://history.siggraph.org/learning/double-negative-presen...
Is Intersetllar the movie where Matthew McConaughey turned into a bookcase?
No not that one.
Not that one either.
TLDR “A typical IMAX image has 23 million pixels, and for Interstellar we had to generate many thousand images, so DNGR had to be very efficient. It has 40,000 lines of C++ code and runs across Double Negative’s Linux-based render-farm. Depending on the degree of gravitational lensing in an image, it typically takes from 30 minutes to several hours running on 10 CPU cores to create a single IMAX image. Our London render-farm comprises 1633 Dell-M620 blade servers; each blade has two 10-core E5-2680 Intel Xeon CPUs with 156GB RAM. During production of Interstellar, several hundred of these were typically being used by our DNGR code.“
Small VFX shops do typically render on GPUs nowadays, but the high end is still dominated by CPU rendering.
NVIDIA H200 GPU has 141 GB of memory, and the AMD Instinct MI300X GPU has 192 GB of HBM3 memory.
Besides, in the current climate I don't think VFX companies would be eager to bid against AI companies for access to H100/H200s, the VFX companies don't have infinite venture capital goldrush money to burn...
I can't remember the actual size of the simulation in terms of number of particles, but it was using something like 2 150TB fileservers to store it on.
With NVME and cheap ram, it would probably at the point where it'd be useful on a GPU.
https://www.shadertoy.com/view/lstSRS
No idea if it's doing the same simulation though.
Jokes apart — this is the best SciFi movie ever! It’ll become one of these movies one’s got to rewatch every 5y or so.
~26 2/3 frames per minute, not bad
If the planets they visited had much lower gravity then earth this may be possible, but this wasn't noticeable or talked about. And even on Mars you need much more fuel to get into orbit than could be stored in their tiny ship.
Heck, you could even ignore the jet aspect, and go for full rocket. (An ICBM only weighs around 50 tonnes, after all.) Getting the heavy thing into orbit, though, requires proper, multi-stage rockets.
A jet engine requires atmosphere. Spaces requires no (or very little) atmosphere.
The edge of space is usually defined at 100km (the Karman line).
The highest altitude reach by an air breathing jet is around 37km. https://en.wikipedia.org/wiki/Flight_altitude_record
A scramjet might get to ~75km according to: https://aviation.stackexchange.com/questions/44837/what-is-t...
Liked the killer space monkeys though. ;-)
We have a plague on Earth and doing an interstellar travel is the solution?
I was very disappointed. I’m OK with a few plot holes, but this is so fundamental to the story that the movie can’t stand on its own.
The issue on the literal level of the narrative is that if you have no way to control the infection on earth, there's no reason to suppose you won't bring it with you to the next planet.
Cooper sobbing as he listened to years of messages from his family? I wish I could've written that. The twist where Mann activated his beacon because he didn't want to die alone? Solid.
We need the gravity equation to get off Earth because the crops are dying? Love can escape a black hole through the tesseract? This is just patent technobabble nonsense.
Where _Interstellar_ fails is in exposition. Love is the most powerful force in the universe? The Fifth Element pulled that off by not pretending to be serious. Anything is permitted in science fiction, but once you commit to explaining it, you need to make sense.
Take _Star Wars_ for example. You don't need to know how the Force works. There's no suspension of disbelief because you've bought into the genre and its tropes. The Force is everywhere, Luke is atuned to it. We don't need to know anything else mechanixally. Then they rolled up with midichlorians and reduced the whole thing to jibberish.
_Interstellar_ sets a hard sci-fi expectation with a visually accurate black hole before losing the train of thought. Imagine if _Lord of the Rings_ spent 20 minutes discussing how the plate tectonics of Middle Earth gave rise to Mount Doom before veering off into the history of the eagles. It gives your audience whiplash.
The small craft could launch from a planet with twice the Earth’s surface gravity, make it to escape velocity, and meet the Endurance?
Why would they need better propulsion than that to leave the planet and build space habitats?
Under these circumstances (only a couple years left until starvation begins & no existing industry) it seems absolutely realistic that the government is able to throw together one hailmary in secret, without also being able to scale production up to save most people before they starve. Remember, the goal of the mission wasn't to measure the data they need to finish their equations, it was to send out one single ship that could continue human civilization on an inhabitable planet.
Had the future humans not intervened (by placing the black hole near inhabitable planets), humanity on earth would have died.
Not only a black hole, a supermassive Kerr black hole with relativistic rotation at its equator.
They could have done a better job at the planets themselves though - Most of them were pretty useless for terraforming.
While I'm sure the US government would trade immediate comfort for long-term survival of the species, I don't think other governments would have the same priorities. I'd assume China would be one of the first to build multiple O'Neil cylinders with carefully controlled environments and as little Earth-originated biomass as possible (to contain the blight).
and then just a lame story culminating in "space ghosts"
for me, arguably most disappointing movie ever
It'd more like a producer going:
"is it done yet?"
"what do you mean it'll take half our rendering budget, I need that for delivery"
"Is it done yet?"
"can you make it look more spacey?"
"no thats too spacey now"
"Why isnt there lens flare?"
"Why does it take two weeks to render, can't you fake it?"