Holy crap that’s amazing! How complex is your model? And are there lots of parallelizable parts like filters or is it recurrent?
We managed to get up to 10x for very low resolutions (160) for a resnet101 but it usually plateaus for high resolutions (above 896x896) at a 1.7~1.9 speed-up. Although using Int8 gives even higher speed-ups (~times 3.6 for 896x896 input), for some tasks it degrades the performance too much.
I will definitely try your setup :)