Idk about using these libraries, but its almost impossible to find generic graph libraries that aren't designed around either ML or alternatively scheduling batches. One such example is my own, https://github.com/timkpaine/tributary
But this is a completely different kind of graph. The graphs being discussed here are differentiable DAGs of mathematical computations.
That said, Tensorflow and Pytorch are both very much general purpose numerical computing libraries. You don't have to use them for neural networks.