Doc Bailey had custom software. I had the pleasure of working with it on a bunch of films in combination with some rendering software I wrote.

Doc Bailey passed away quite some time ago now though and he never released the source code to my knowledge: https://vvvv.org/blog/richard-doc-bailey-(spore)-passed-away

Looks a lot like https://www.electricsheep.org/

They're IFS fractals, I recommend either Apophysis 7x or Chaotica for creating them.