Node-based a/v composition: programs as graphs and graphs as compositional tools | Hacker News Reader

Node-based a/v composition: programs as graphs and graphs as compositional tools

github.com

59 pointskousun1219 comments

If someone's interested in moving the node-based media software status quo, I've been developing ossia score for the past few years : https://ossia.io

A big difference with other similar environments, is that the dataflow graph evolves dynamically based on a timeline which is useful for interactive shows, etc. I've recently been working on doing the same for shader-based stuff : https://twitter.com/jcelerie/status/1211354889873510400

Intriguing. Very surprised I've never heard about this before, and refreshingly different from most other tools in this field. I look forward to trying it out.

He didn't mention a few other big ones with good demo versions: Nuke, Touch Designer and Digital Fusion (much more awkward, but free).

Figuring out why programs like these work well and what gaps are between them and general programs is interesting because many of the reasons come down to pragmatic workflow and not technical limitations.

They all operate on a very limited set of data types (just images, images + geometry, etc.) and don't have to deal with state, branching, or loops. Results can be viewed easily at each stage and each operation can be isolated and iterated with real time interactivity.

This project tries to use message passing nodes as a backbone and data flow as the heavy lifting to get to visual C++ programs:

https://github.com/LiveAsynchronousVisualizedArchitecture/la...

> Digital Fusion (much more awkward, but free)

Small nitpick, only applies to Fusion 9 and below. Fusion 16 only comes in the Studio variant, otherwise you need to use DaVinci Resolve 16 to access the embedded Fusion for free.

> Figuring out why programs like these work well and what gaps are between them and general programs is interesting because many of the reasons come down to pragmatic workflow and not technical limitations.

In the case of Nuke, the pragmatic workflow is a large part of it, but not the only one. When it comes to digital compositing, it is very easy to get up and running with layer based solutions quicker (such as Adobe products), but they hit a wall very quickly on memory usage, process duplication, and general difficulty of navigation and organization as the composite becomes more complex.

Nuke is in a different class of its own, being developed originally at Digital Domain (a VFX studio) before eventually sold to Foundry (then The Foundry). Nuke's success was a technical one, being far more memory aware and efficient (a scanline renderer and processor), as well as catering directly to the user base it served. It is still the only specialist compositing tool that has support for modern deep data, has proper handling and support for multi-layer/multi-part EXRs, and has a clean and proper handling of channels and layers throughout the comp. It has some modernization to undergo nowadays, but it is still the most trusted tool in VFX.

Software in computer graphics to note as being primarily node based that expose that directly to the user:

  - Autodesk Maya (is node-based)
  - SideFX Houdini (formerly Prism)
  - Foundry Nuke (originated from Digital Domain)
  - Foundry Katana (originated from Sony)
  - Substance Designer
  - Isotropix Clarisse BUiLDER
  - Natron
  - Blackmagic Design Fusion (an in-house tool to Eyeon Fusion -> Blackmagic Design)

Other tools that integrate node based workflows into their application:

  - Autodesk 3DS Max
  - Foundry Mari (originated from Weta Digital)
  - Maxon Cinema 4D
  - Blender (personally unsure of Blender and being nodal-based)
  - Blackmagic Design DaVinci Resolve

There are a smattering of others I haven't gotten the chance to use, from other imaging tools to those specializing in tracking (e.g. PfTrack) and in-house non-commercialized tools.

The most trusted tool since Apple shitcanned Shake, that is.

It's too bad that Shake couldn't be open-sourced. Several big post facilities (ILM, Weta, and others) bought the source code and have probably done great things with it.

No one really did much with it. Nuke is actually much better than shake by a wide margin, using shake feels like going back to the stone ages due to interactivity.

I wrote plugins for Nuke 12 years ago, I was just trying to give an interesting introduction to programmers.

From a creative perspective the modular/node-based way of thinking has always been a niche, having seen many attempts such as this to place it into a broader model of creativity (including my own [1]). Its telling that the enduring musical instruments of the electronic age have been relatively fixed-function (Minimoog, Yamaha FM synths, 808/909) with constrained options for patchability, rather than modular synths highlighted by the article. The metaphor is further strained by modular synthesis' lack of basic affordances a programmer would expect, such as a memory model, data types, composability/reusability of "code", etc.

These issues translate to digital node-based systems- dealing with structured data in e.g. Max or TouchDesigner becomes an exercise in patience, and their mechanisms for composing and reusing code are somewhat baroque as well.

The other major creative limitation of flow-based programming (and in fact, most computational art systems) is the absence of a timeline (as mentioned by another commenter). This is a showstopper for many creative pros Ive talked to. But it grazes on topics that are interesting purely from a computer science perspective- imagine if you could freely jump to any point in a program's execution?

To be clear, having spent my doctoral studies in this space I find these to be interesting problems to solve rather than indictments of node-based programming as a whole. Im interested to see how eternal and other efforts mentioned in the comments here tackle these and other challenges.

[1] https://purl.stanford.edu/mf249vj6694

Agree on most of this; most systems I've used so far feel a bit clunky (Houdini has by far been the most sophisticated system I've worked in), and my sense is that if it had more mindshare / attention, the paradigm could actually be pretty fluid. Using most node-based tools today (including eternal), I get the admonition that it's like using text edit to write python compared to a pro at using pycharm.

The timeline thing is really key. Houdini does it in a sort of clever way where "time" as a variable is exported globally to any node that wants to use it, and the timeline feature is a consequence of using the time variable in any of your functions. i.e. it's not jumping you to a point in the program's execution, but rather acting as a controller for all time-dependent functions. sort of like exporting a time uniform to graphics shader

Agreed- a lot of the issues people bring up with node-based programming have promising solutions, its just a matter of exploring the space and finding successful implementations.

As far as timeline, I also think the digital audio workstation model is interesting- modules export parameters that can be assigned to an automation channel and dragged/tweaked visually on some hierarchy of timelines.

Flow-based, node-based, visual - whatever you want to call it, it is programming and needs one to think like a programmer for anything but the most simple tasks.

In fact - it can be more complex than "actual" coding in some ways as the pallette of abstractions is often more limited - or the interface creates hoops to jump through that a text-based format wouldn't.

(I do like jumping between text and visual node-based representations however. Learnability and discoverability are an advantage of the latter. And it fosters a different style of tinkering.)

With regard to timelines - I can think of several ways a timeline would work with a graph. It can simply be another input for any exposed parameters. Or it can be a node in it's own right.

this is a great project, i am interested to see more competitors to pd, vvvv, and max matthews little program. a lot of great work has been made on tools like this.

check out touchdesigner, it has the ability to go deep into GLSL and python but is also quite approachable. i feel like it is generally regarded as a node based graphics programming environment, but would argue that it is really just an arbitrary real time data handling system that happens to have a lot of built in graphics nodes.

I've never found pd nor max to be productive.

why not? i use neither but they both have rich histories as audio programming tools. autechre are/were heavy max/msp users at the forefront of IDM, max for (ableton) live is a popular thing, and i have several friends who write pd things that then get run on their eurorack modules.

This eternal project is crazy crazy crazy good.. Achieved in the space of only a year.. This is incredible.

Beautiful. Really.

thanks! appreciate it. would loved to have spent some more time on it / made it more of a primary thing for me, but i suspect it's a sort of all or none kind of work && i hope someone can get a team together and do a version of it for real

A git or otherwise nice version control with diffs and such for a visual language would be killer.

For a flow-based programming DSL, check out FlowNote

https://emblem21-opensource.github.io/flownote/docs/02-featu...

Reply on news.ycombinator.com