Unfortunately, no one has the time to do that (or let someone do it) after the problem is "solved", so over time the "rebuild from scratch" approach just results in a loss of actual troubleshooting skills and acquired knowledge --- the software equivalent of a "parts swapper" in the physical world.
I get the desire to troubleshoot but priority 0 is make the system functional for users again, literally everything else can wait. I once had to deal with an outage that required we kill all our app servers every 20 minutes (staggered of course) because of a memory leak while it was being investigated.
What numbers went into this calculation, to get such an extreme result as concluding that getting it up again is always the first priority?
When I tried to estimate the cost and benefit, I have been surprised to make the opposite conclusion multiple times. We ended up essentially in the situation of "Yeah, sure, you can reproduce the outage in production. Learn as much as you possibly can and restore service after an hour."
This is in fact the reason I prefer to keep some margin in the SLO budget -- it makes it easier to allow troubleshooting an outage in the hot path, and it frontloads some of that difficult decision.
You can probably see where this is going. The workers had some problem where they would bog down if left running too long. Causing the queues to back up and indirectly causing themselves to eventually be culled.
Never did figure out why they would bog down. We just ran herky jerky like this for a few years till I left. Might still be doing it for all I know.
So you just automatically replace the instances after a certain amount of runtime and your problem is gone.
Is this really a prevailing attitude now? Who cares what happened, as long as we can paper over it with some other maneuver/resources? For me it's both intellectually rewarding and skill-building to figure out what caused the problem in the first place.
I mean, I hear plenty of managers with this attitude. But I really expect better on a forum called hacker news.
If it happens extremely rarely (like, once every 6 months) or it’s super transient and low impact, we kick it and move on.
If it starts happening a 3rd or 4th time, or the severity increases we start to dig in and actually fix it.
So we’re not giving up, and losing all diagnosis/bugfixing ability, just setting a threshold. There’ll always be issues, some of them will always be mystery issues, so you can’t solve everything, so you’ve got to triage appropriately.
I don't really fault the junior dev who does that. They are just following the "I don't understand something, so just start over" paradigm set by seniors.
If you have proper observability in place then you can do your diagnosis without affecting your customers.
Plus, at the same time successful diagnosis is also the kind that can have the most dramatic effect on your customers.
In a positive way.
A special case of this is git rebase after which you "can" access the original history in some obscure way until it's garbage-collected; or you could clone the repo before the merge and then you can access the original history straightforwardly and you decide when to garbage-collect it by deleting that repo.
I would never ridicule anyone for your approach, just gently encourage them to spend a few mins to grok the ‘git reflog’ command.
- Commit any pending changes.
- Make a git tag at your current head (any name is fine, even gibberish).
If anything “goes wrong” you can rollback by simply doing reset hard to the tagged commit.
Once done, delete the tag.
Making a complete “backup clone” is a complete waste of time and disk space.
The two primary source code management activities developers use are versioning of source code (tracking changes which happened over time) and the other being synchronisation of code with other developers.
One of Git’s differentiating strengths is it being decentralised, allowing you to do many operations in isolation locally without a central server being involved. You can then synchronise your local repository with an arbitrary number of other copies of it which may be remote, but you may need to rebase or merge in order to integrate your changes with those of other developers.
Git is more like a local database (it even allows multiple local checkouts against a single common “database”) and it only occasionally “deletes” old “garbage”. Anything you do locally in Git is atomic and can always be rolled back (provided garbage collection hasn’t yet been performed).
Although I’m comfortable enough with using the reflog to rollback changes (I’m also skilled enough in git I haven’t needed to in many years), it’s not very user friendly, it’s essentially like sifting through trash, you’ll eventually be able to find what you lost (provided it wasn’t lost too long ago), but you may have to dig around a bit. Hence my suggestion of tagging first, makes it easy to find it again if needed.
I have very limited Linux experience and have no recommendations on your other question.
So... more generally, your idea with replacing VMs is rather naive when it comes to storage. Replacement incurs penalties, s.a. eg. RAID rebuilds. RAIDs don't have the promised resiliency during rebuild. And, in general, rebuilds are costly because they move a lot of data / wear the hardware by a lot. Worst yet, if you experience the same problem that caused you to start a rebuild in the first place during the rebuild, the whole system is a write-off.
In other words, it's a bad idea to fix problems without diagnosing them first if you want your system to be reliable. In extreme cases, this may start a domino effect, where the replacement will compound the problem, and, if running on rented hardware, may also be very financially damaging: there were stories about systems not coping with load-balancing and spawning more and more servers to try and mitigate the problem, where problem was, eg. a configuration that was copied to the newly spawned servers.
But consider that the case for older companies, where it was more common to deploy several systems, often complex ones, onto the same node. You will also cause outages to system x, y and z too. Maybe some of them are inter-dependent? You have to outwhey the consequences and risks carefully in any situation before rebooting.
At least as I read it, this contains the assumption that that‘s not how you deploy your applications
Yeah we do this? It doesn’t pose an issue though. Cordon the node (stop any new deployment going on), drain it to remove all current workloads (these either have replicas, or can be moved to another node, if we don’t have a suitable node, K8s spins up one automatically) and then remove the node. Most workloads either have replicas spare, or in the case of “singleton” workloads, have configs ensuring the cluster must always have 1 replica available, so it’s waits for the new one to come up before killing the old. Most machines deploy and join the cluster in a couple of minutes, and most of our containers take only like, 1 or 2 seconds to deploy and start serving on a machine, so rolling a node is a really low impact process.