I can speak to 1.) a bit as I have done just that for a client of mine. By default, they're using Zencoder/Rackspace to encode/store their videos and usually get one-off videos from their customers. However, sometimes there are requests to encode 2.000+ videos at once which gets prohibitively expensive in their constellation.

So we basically started out with a very naive solution: A Docker image with ffmpeg and access to a shared Redis DB used for job coordination running on a somewhat beefy server from Hetzner (beefy compared to regular VPS offerings, something in the 40€ range with 16 or 32gb of RAM I think). This turned out to work surprisingly well so we built a small API around that mirroring parts of Zencoder's API. This way, we can switch between both services on a per-job-basis and only need to change the encoding service's endpoint URL.

The main thing that will get you (if you do input validations) are the video formats. I've tried to get an authoritative list of valid MIME types beforehand, but still people managed to somehow send us videos with weird encodings that I'd never heard of before. Other than that, the usual caveats regarding self-hosted services (availability, failover, backups, etc.) apply but have so far not been much of a problem to us (especially since we can just go back to Zencoder when something breaks and needs to be done fast).

Feel free to drop me a message (mail is in my profile) if you have any questions.