Woah. Heroku is nice, but doesn't $6400/mo seem insane for a quadruple extra large ec2 instance? That's an impressive margin.
We've moved some stuff off Heroku but not because of cost, most recently we discovered they have a tight max on scala apps and it was cheaper for us to put it on aws directly, rather than code around it on Heroku.
Let's say we have 5 engineers and that kind of training costs us 1 hour per eng/per week. That's 20 hours a month, which is easily more than $1k/month and the cost grows as the team expands. This is before you account for the losses of having your own engineers on call for issues and setting up early warning systems specific to the database etc.
Hardware aside - thats largely moot on both Heroku and AWS - the 5x cost of the server will very quickly outstrip the cost of a good developer/operations guy. It seems like a good idea - I initially just throw money at a problem too! - but at some point it does not make financial sense, which is what I was getting at.
Note that we don't have every engineer on staff capable of bringing up a new server/rebuild a damaged mysql replication setup, but we do have engineers that can:
- ssh (or attempt to ssh onto) a dead instance
- tail a log
- check disk space, memory usage and cpu load
- check if something is running
- restart something that just randomly stopped
As far as early warning etc., that does take time, but it's not the big deal it appears to be.
EDIT: I can't math, and 80 + 10 = 90, not 100. Good thing I'm not a data scientist :)
EDIT: munin does also ship monitoring plugins for MySQL but since I have never used them I cannot vouch for them. The general health monitoring (disk usage, CPU usage, SMART status, inode usage, ...) of the machines provided by munin is also probably more valuable than the database specific monitoring.