>the first baseline was to freeze the tensorflow graph and just run it directly on the pi. this works without any problem, it's just the pi can only do 1 image / second :/

I wonder if this was done using full blown tensorflow, which might be leaving a lot of performance on the table. There are currently some efforts to improve the efficiency/optimization of getting a network to run well on the modest hardware of a pi e.g., tensorflow lite: https://www.tensorflow.org/mobile/tflite/ tvm: tvm.ai