Hacker News Reader

> It is not meant to be used inside a DCC app, as-is.

> If you import data of this complexity into a DCC app your workflow is broken/you are doing something wrong.

are you sure you're not just saying this because years of software being slow made people cargo-cult workarounds around the issue which are now assumed to be "the way" in the field ?

The software is fast where it's needed - it makes the right trade-offs. There's little point in building IDEs capable of handling terabyte-sized source files, because just having a terabyte source file is ridiculous.

Scenes like Moana are not intended to be handled as a whole, they are assembled from little pieces - semantically independent and assignable to different people with different skill sets. It's not the matter of software performance, it's the matter of process organization.

At some point, the little pieces need to be assembled, reviewed and finally rendered. Why wouldn't you do that with the DCC application rather than with specialized, limited tools?

Indeed, you often do that "somehow" inside the DCC.

But again: the DCC doesn't see the high detailed version of that geometry in a proper pipeline. It only sees a proxy. See my detailed reply above.

Only the renderer does see the final geometry and even that may be generated procedurally, on demand. I.e. stuff like fur, particles or the like is often generated on the fly since caching (and then reading) this data is not only impractical but also may be slower.

Think a terabyte of fur data for a horde of fluffy monsters on screen (per frame, of which there are 24 per second of film at least) that needs to be distributed to each blade on a render farm or, worse, machines in the cloud (again and again since each frame goes through many revisions in lighting/rendering before its final).

Computing the fur from a few hundred thousand 'guide hairs' on each character, on the fly, is a better approach.

The Blender OBJ loader was incredibly inefficient. Blender used something like two orders of magnitude more RAM than necessary, so even a small model would take gigabytes. It was basically useless for any model of even modest complexity. Working around its limitations was dozens of hours of effort when I was importing and rendering my own procedurally-generated models.

I was seriously considering rewriting the Blender OBJ importer myself. Ultimately, I ended up writing an OBJ to Alembic converter, since Blender handled those more efficiently.

> are you sure you're not just saying this because years of software being slow [...]

Yes. And I don't know where to even start explaining this. Don't get me wrong.

"Simply put": there is no use case for this. Moana e.g. has gazillions of vegetation geometry instances. Why would you have those inside a DCC? Pebbles on the shore, etc. etc.

VFX pipelines are specialized. An animator does not care if there are 10 million hairs on their furry character while they do the animation. A lighter only cares to see the fur when it's rendering, not while operating the light rig in the DCC. Modern pipelines have instant feedback/viewport rendering where proxy geometry is swapped out for "the real thing" by the renderer (or a custom plugin inside the renderer) on the fly.

The result, inside the renderer, is something like the Moana dataset. Hence its publication. It's a stress test for renderers, not DCCs.

> "Simply put": there is no use case for this.

there can't be for now, because artists couldn't realistically do this. I've been working with artists for, what, close to ten years now and I can guarantee you that their use of the software permanently fills the performance increase that such software gives. That's like saying, why would a music sequencer need to support more than, say, 100000 tracks, which sounds ridiculous until you see someone actually coming up with a fun music score that leverages generative scripting to create very cool pieces

The performance increase is eaten by scene complexity.

This is and has been the norm since the advent if CGI in blockbuster movies, in the 1970's.

For every order of magnitude that hardware gets faster (and/or software gets 'better', i.e. using better parallelization etc.), image complexity in your average shot increases by an order of magnitude.

So yes, what artists are doing is more complex but relatively so. I.e. it doesn't invalidate anything I and others wrote about the case in point – the Moana asset, how such assets are built and what part of a VFX pipeline reasonably ingests them.

"The point" was to find a large project to test import performance, and it sounds like they hit that point dead center.

What is the point of importing a dataset of this complexity if you can't also work with the data inside the DCC?

Try the thousands of tools and plugins available for Blender with such data and see what happens. Good luck.

I guess my point is: you need to also "fix" any of those that crash/hang/are too slow to make this worthwhile.

Just to be clear again: awesome they fixed the bugs that lead to crashes with this data. Not sure if that large a dataset was required at all to do that though.

But the speed improvements? Do they matter for your average OBJ? And if no one ever imports data of the complexity of the Moana dataset, because they can't work with it afterwards anyway, any speed improvement that is not felt in the average use case is a nice engineering exercise, foremost.

There is a reason why most commercial DCCs, too, struggle with data sets of this size and no one has ever "fixed" this. ;)

> What is the point of importing a dataset of this complexity if you can't also work with the data inside the DCC?

My understanding is, that you need to reproduce the rendering bug which crashes blender to be able to fix it. And being able to reproduce it, needs to be fast. Even if you have a smaller scene which would trigger this bug, without the optimisations it would take a lot of important time in the feedback loop. Now you have a workflow which crashes the rendering in less than 2 minutes.

See this reply from someone else: https://news.ycombinator.com/item?id=32190577

See the reply to that:

https://news.ycombinator.com/item?id=32191173

>At some point, the little pieces need to be assembled, reviewed and finally rendered. Why wouldn't you do that with the DCC application rather than with specialized, limited tools?

See the explanations in my other replies on this topic.

It is impossible to hold a typical VFX scene in RAM to start with.

Even freelancers doing sim/FX work now have at least 128GB RAM and this is often just enough for proxy work that still gets expanded at render time.

I.e. consider the possibility that your worst estimate of how complex this data could be is off by 1-3 orders of magnitude.

And for your example: the person who signs this off is the VFX supervisor. They don't sign off anything but final frames.

I really don't understand your point at all. You might be right, the use case doesn't exist yet, maybe never. But this never was the point of the blog post.

Making the application wait extremely long or even crash by importing _something_ is a bug in my understanding. Why shouldn't it be fixed? Why shouldn't developers improve performance and blog about it, so other devs learn from it?

It's not about the Moana scene, that's just the test case, so OP has a valid benchmark with human comprehensible durations. The scene could be anything that is smaller and it will be imported faster now.

He sped up parts of the blender pipeline, someone will no doubt benefit.

He does address that the crashing also needs to be fixed but that he doesn't want to be the guy doing that.

It’s a stress test. It doesn’t have to be reasonable. This is what you do for performance improvement, you push as hard as you can to find the worst actors and fix them.

Performance improvements like this expand the horizons of what is reasonable to do.

There’s a circular logic problem where you argue that performance problems shouldn’t be fixed because nobody actually does this, but nobody does it because of performance problems. No, this effort didn’t solve every performance problem required to do a thing, but did it solve some of it.

“We didn’t do everything so we should have done nothing” is not how you build a performant product.

I disagree, the point was to show how object importing scales and he used an extreme example to make the point. He also gives an example of duplicating 10k cubes in Blender (something not out of the ordinary) also sees a massive speed up.

This kind of thing can help understand other bottlenecks in software like Blender as its users and developers get more ambitious as the software grows in capability.

Idealistically wouldn’t one like film makers to be able to work ‘in world’ decoupled from technical limitations?

I've seen much the same problems when trying to programatically construct a large number of objects using the (low-level) Python API. If this kind of optimization improves throughput of those APIs that would be a major enabler for work that I want to do.

Otherwise I'm side-tracked looking for relatively awkward SIMD-style programming (e.g. Geometry Nodes) as a workaround for "CRUD" operations being too slow. Can't be just me...

It doesn't miss the point at all. It's a large data set and they used it to improve a function of Blender. There's nothing that says it only has to be used to test renderers.