A lot of what you need is already written, you just need to find the right libraries and write the bindings. From my encounters with Python ML it seems like "just use Pytorch" is a bit like "simply walk into Mordor".