PilotPDG Procedural Dependency Graphs from SideFX

To add a bit of background, this is a new 'Top' (for 'Task Operator') context in Houdini, a new class of licensing from SideFX and eventually it will be a new standalone application as well.

Currently you can try out PDG by installing Houdini 17.5. You can get apprentice (a free, non-commercial Houdini license) which is only limited in what resolution you can render, or generate render-intermediate files for (in other words it stops you from using a render farm but does not stop you from evaluating or learning almost anything about Houdini).

PDG is designed to be able to automate any processing task using a network/graph of nodes where the connections describe the dependencies between tasks, including complex dependencies. It can also be used to distribute the tasks to a renderfarm/network of computers.

PDG ships with nodes that can call on imagemagick, as well as ffmpeg, maya, etc and has readers for json/css/xml/etc (to help with turning random files into sets of tasks). The main language you would use with PDG would be Python.

There are several examples where it has been used to generate/prepare Neural-Net training pairs to feed into pytorch, and also to co-ordinate hyperparameter wedges and training rounds followed by network performance evaluation. See the Houdini17.5 launch video(s), the section where they make a terrain generator that is 50,000x faster than solving an erosion model (that also ships with Houdini).