You're probably going to get much better performance if you convert your TensorFlow model to CoreML (use tfcoreml). Also has the advantage of not needing to compile TF for your phone yourself, and distribute it with your app.
https://medium.com/jigsaw-xyz/realtime-image-recognition-in-...
(I wrote and maintain Keras-RCNN. It does not compile to CoreML. Yet.)