Very cool! How is the performance? R-CNN used to be much slower comparing to YOLO or SSD; FCN seems to be very fast as well though requires a lot of GPU memory. Can that your version be used for realtime semantic segmentation?
With that said, there are a lot of things you could do to make this much faster. For example, use ResNet50 instead of ResNet101. You can also reduce the number of anchors or the number of proposals to classify, and that should improve performance significantly at the expense of a little loss in accuracy.