So you're not even using ffmpeg? It's used by Dropbox and many others - https://tech.dropbox.com/2014/02/video-processing-at-dropbox...
I'd imagine that'll be the biggest bottleneck, especially if you intend to transcode raw video with data rates of > 1GB/minute