5,556 karma · joined July 10, 2013
https://about.gitlab.com/blog/2017/02/10/postmortem-of-datab... - thousands of GitLab projects permanently lost. A stark reminder of the importance of database backup testing procedures, because even multiple layers of backups can fail in a black swan event.
TL;DR: I love being an engineer in the Bay Area, but we truly are a bubble.
Medicine - COVID vaccines.
Science - global warming.
Unfortunately, humans and their politics are intertwined into more aspects of life than one might deem rational.
Having seen a much smaller set of Cassandra nodes used to store billions (rather than trillions) of records, I can say that Cassandra was definitely a total PITA for on-call, and a cause of several major outages.
I use AWS SES for transactional emails (mostly send, but making use of the receive capability as well). Nice and cheap. And finally using MailerLite for marketing emails. Just surpassed their free tier so may look again, but not too bad pricing-wise.
One advice I've seen is not to mix transactional and marketing providers (and ideally domains). If you get a high incidence of unsubscribes or "this is spam" responses, don't want your app to be dead in the water for transactional stuff if the marketing provider suspends your account temporarily (can happen even if you're legit).
To be clear, I'm not saying you should be working extra hours to do this. If that's required, either your manager is not reasonable or you're likely not there yet and if I were to promote you, you might burn out soon.