Now that is definitely interesting. And you have some notion of "differentiability" for all of your various sources, sinks, and transforms?

That said, Tensorflow and Pytorch are both very much general purpose numerical computing libraries. You don't have to use them for neural networks.