What we learned after I deleted the main production database by mistake

If you can't recover in peace time, what makes you think you can do it in war time?

there's an interesting piece on the web about how HD is not necessarily an egg, that that idea comes from an illustration added long after the nursery rhyme was well established.

https://literature.stackexchange.com/questions/1489/how-do-w...

Well maybe the people that enforced those processes, wouldn't you think?

Instead it comes across as a veiled condescension. It doesn’t impress.

At some point a company needs to adopt practices like Netflix's chaos monkey or Google's DiRT (disaster readiness testing) to purposefully excerise continuation of business plans as well as recognize the effort that is required to keep things running. Otherwise other incentives will drown out any intrinsic motivations individuals may have to improve reliability.

But all those processes and certifications are the kind of things people bemoan in big corporations. And start-up don't have the resources nor time to do it, even if the sooner it's implemented the cheaper it is. That's IMO where incubators could add some value: have a team of people whose job is to setup and follow those processes for your smaller start-ups.

From TFA, next sentence after the one you quoted:

Also, that database was a read model and by definition, it wasn’t the source of truth for anything. In theory, read models shouldn’t have backups, they should be rebuilt fast enough that won’t cause any or minimal impact in case of a major incident. Since read models usually have information inferred from somewhere else, It is debatable if they compensate for the associated monetary cost of maintaining regular backups

The article discusses how they executed an actual recovery process from the other data sources, but it took longer than it should have (6 days), so...

We ended up with a mix of the two. We refactored the process to a point it went from 6 days to a few hours. However, due to the criticality of the component, a few hours of unavailability still had significant impacts, especially during specific timeframes (e.g. sales seasons). We had a few options to further reduce that time but it started to feel like overengineering and incurred a substantial additional infrastructure cost. So we decided to also include backups when there was a higher risk, like during sales seasons or other business-critical periods.

There you go.

If I calculated this right the time they mention comes down to 30 items per second. Which is maybe not unreasonable for something that queries a whole bunch of services via HTTP, but is kinda ridiculous if you compare it to directly querying a single RDBMS.

You could probably fix this by scaling everything horizontally, if that is possible. But the real solution would be as you say to have bulk processing capabilities.

That's what blameless post mortems are supposed to prevent. The only valid things to consider are questions like "is it a cost-effective prevention?", "what is the timeline of implementing this prevents / how should we prioritize?", and "does it meaningfully overlap with other changes that would prevent this?". Things like "well person A is new and that's why this happened" is not an acceptable position. Now of course, people can be not honest / not self-aware enough and are coming from that position without stating it anyway. That being said, that's why you try to focus on the objective evaluation of the recommendation and nothing else - that tends to make the bias irrelevant. Any EM/tech lead worth their salt would instantly say "is it actually important that staging and dev databases share creds"? Certainly prod should never. The other follow up to track down would be "if DNS flushing is so important when changing hosts, how have we automated it so that it happens automatically before every script that's run". I would also recommend to get rid of (ab)using the hosts file immediately and switch to explicit configuration files that you have to specify (perhaps automatically picking up "<username>.config" as a default so that scripts only ever run on your dev zone by default & locking down production.config so that only CI has the ability to touch that).

I’ve now taken 3 companies from horrid MVP (literally just got seed funding) to something a team enjoys working on. Each time I’m amazed that the app went without issues long enough to get funding (and long enough for me to refactor the whole thing.)

We want an internet with less ads, but good writers deserve to get paid. They can get paid via Medium (though how much, I don't know) through subscriptions. Is that the worse than ads or newspapers?

If from the comments here, or wherever is referring me to an article, I get the impression that it will be interesting/useful to me, I'll maybe go look, if not then I won't. Or sometimes the discussion the link starts is enough such that I get useful references of things to look at in other places instead (or just learn what I might want to from the discussion directly). Heck, sometimes just the subject is enough to start searching for other references.

Postman doesn’t know that sending a single DELETE request to that URL will delete 17 million records.

Arguably, REST interfaces shouldn’t allow deleting an entire collection with a single parameterless DELETE request.

`requests.get(url)` is a lot harder to mis-type as `requests.delete(url)`.

At $dayjob we would sometimes do this sort of one-off request using Django ORM queries in the production shell, which could in principle do catastrophic things like delete the whole dataset if you typed `qs.delete()`. But if you write a one-off 10-line script, and have someone review the code, then you're much less likely to make these sort of "mis-click" errors.

Obviously you need to find the right balance of safety rails vs. moving fast. It might not be a good return on investment to turn the slightly-risky daily 15-min ask into a safe 5-hour task. But I think with the right level of tooling you can make it into a 30 min task that you run/test in staging, and then execute in production by copy/pasting (rather than deploying a new release).

I would say that the author did well by having a copilot; that's the other practice we used to avoid errors. But a copilot looking at a complex UI like Postman is much less helpful than looking at a small bit of code.

One place I worked we had detailed step by step checklists with ready to copy paste commands for patching server. The Windows process was forever drifting and getting out of date while the Linux process was being kept up to date.

The difference was that the Windows admins were all morning people who enjoyed showing up to work at 7am or earlier if they could justify it and just doing it from memory. While the Linux team were all night owls who preferred to roll out of bed 5min before the update schedule, remote in, execute the process with as much automation as possible in the hope of getting some more sleep before having to got to work.

Basically the Windows process was only ever actually followed when someone outside of the Windows admins had to cover for them and the Linux process was used every time by the Linux team.

Testing / separation of dev environments is almost black magic.

From the article, it sounds like it would have helped to formulate the correct command in a non-production environment before moving to production

I agree. And luckily that was the case for them.

You can use a tool like elasticsearch-curator or even cron to manage running backups or use the built in scheduling (snapshot lifecycle management)

In other words, it only has to be good enough for a few days (ideally - hours).

Also, what happens if somebody nukes the event table? How is your precious event sourcing supposed to save you then, huh?