Ok, enough AWS outages to say I'm tired of hearing about low end stuff being flaky.
Meanwhile, I currently have a gig to work on a video service which features a never updated centos 6, an unsupported python 2 blob website, and a push to prod deployment procedure, running a single postgres db serving streaming for 4 millions users a month.
And it's got years of up time, cost 1/100th of AWS, and can be maintained by one dev.
Not saying "cloud is bad", but we got to stop screaming old techs are no good either.
2 - Caching is life. We have 3 layers of caching: cloudflare, varnish, and redis. Most things don't need to be real time. A lot of things can be a month old and the user doesn't care. User need immediate feedback to be happy, but not necessary fresh data.
3 - if you compile nginx manually, you get to use a lot of plugins that can do stuff super fast, including serving videos. You can script stuff in lua that will just skip the backend completly.
4 - mind your encoding. We carefully chose how we encode videos. The ffmpeg parameters are pretty insane, but the space / quality ratio is amazing, espacially on mobile. It takes a lot of time to experiment with those, nobody share them :)
5 - we offload everything we can to cron tasks or task queues. Including, obviously, encoding, screenshooting, etc.
6 - don't hold data you can't lose. E.G: billing. This way you can have a relaxed attitute toward data. If we ever loose a day of business, users will be in a bad mood for a week, but that won't be the end of the world. We don't need a bullet proof system if bullets can't kill us.
7 - give money to ffmpeg and opencv, because damn those things are fast. And good.
8 - servers are hosted accross 2 providers. This way, if one goes down, or decide to stop doing business with us Google style, we have a second one. Happened recently with leaseweb: they shutdown a whole room without offering an alternative.
E.G: votes.
They don't hit the backend on write. We pile them from nginx to redis, then once a day, we aggregate and store on postgres, which the backends will consumme. We just store each vote on localstorage as well so that the user feels like it's real time when they vote, but in reality it's updated once a day. But votes don't affect the money side of our business, so if we lose them one day, it does not mean death.
P.S: yes, posgres/redis/elasticsearch only hold metadata. Videos are stored on disk. There is no docker images, no mircoservices, FS is ext4. Which means with a lot of RAM, the OS FS cache will have most popular videos already loaded and ready to be streamed. Everything is raid 0, so if we get one disk corrupted, you lose the server. But we upload each videos on severeal servers, so when a disk get corrupted, we just replace the whole server. In fact, anything goes wrong on a server, we replace it. It's not worth it to find the root cause, unless 2 servers die in the same way successively.
Regarding the ffmpeg parameters and formats in general: Do you use newer formats too, like AV1 and the like?
At least, that's what I understood.
Another side benefit of being with AWS is when you do have an outage, a lot of other people have outages, and so you sort of blend in with the noise. It's not great to be down, but if you're down and also "big service X" who's also an AWS customer is down, it makes your downtime look less like a lack of competence and more like an unavoidable force of nature.
I worked at a company that's bread and butter was online services (e-commerce SaaS platform, similar to Netsuite) and we had significantly fewer outages than AWS had.
But we had redundancies built in to most things, I'm not saying it was perfect but it worked.
The major difference might be that almost nobody is willing to spend 20% of what they spend on AWS/GCP to have a self-hosted solution.
The reason "cloud is so expensive" is because they're essentially telling you what the price will be and even if they only spend 40% of that on actual hardware and operations: it's more than most companies would invest in themselves.
This is absurd, of course, but it's absolutely true.
When us-east-1 is sufficiently borked the management API and IAM services in all regions tend to go down with it.
Static infrastructures usually avoid the fallout, but anyone dependent on the API or otherwise dynamically created resources often get caught in the blast regardless of region
We didn't take any downtime, but if anything had gone wrong there would have been nothing we could do about it until IAM came back up.
The dependency chains can bite you too. During the us-east-1 outage, a Lambda run by cron-like schedules via EventBridge was itself in an okay state, but the EventBridge events that kick it off were stuck in a queue that was released when the problem was fixed. So if your Lambda wasn't idempotent, and you ran it in another region during the outage, you ended up with problems.
Logging in with root credentials was not possible in any region, and even logging in with IAM creds in other regions yielded an intermittently buggy console
and as is usual with us-east-1 outages management API calls were a complete crap shoot regardless of region
"Let's move our internal app with 50 users to k8s in the cloud." --true story
It's a real shame that the collective world of technology does not properly respect the simple solutions that work.
It is almost funny the dichotomy here. Most technological people "admire" the simplicity, elegance and extensibility of the command line. But tell those same people that the best data store for the solution is a relational database and their nose crinkles up.
Every dependency scrutinized and discarded if possible.
I would probably work for free if someone setup their own on-prem cloud in Tanzu, Open shift, or Ranger and used old school proven frameworks for development.
Working in AWS has been a real shitty experience at these large companies. All the nit picky problems (of which there are thousands) get dumped on devs who are trying to deliver working software.
Agreed. Arguably, not using an existing cloud service is a red flag on any new hires. AWS being the primary, but experience using GCS or Azure are at least viable skills, even if your business is AWS-based.
But the "fad-based-development" meme is not going away any time soon. The incentives in the business are built around it (really! No one want's to work on a boring old relational database solution any more). In the old days it was 4th generation languages, RUP, XML and Function Point Analysis... today it's functional programming, SDKs, big-three cloud PaaS experience or (shudder) block-chain.
I think back to my much younger self, when I thought that technology was something to be mastered to solve real-world problems, and I laugh. Little did I know the real problem to be solved was to figure out how to solve those same-old business problems but with the technology of the season (Kubernetes, GraphQL or ML).
Though I guess there's still probably just lost revenue that could be captured by having better uptime, even if your competitors are down.
Uptime improved rather dramatically after that.