Linux Crisis Tools
brendangregg.com
brendangregg.com
> 4:07pm The package install has failed as it can't resolve the repositories. Something is wrong with the /etc/apt configuration…
Cloud definitely has downsides, and isn’t a fit for all scenarios but in my experience it’s great for situations like this. Instead of messing around trying to repair it, simply kill the machine, or take it out of the pool. Get a new one. New machine and app likely comes up clean. Incident resolves. Dig into machine off the hot path.
"4:10pm the new machine still has the same performance issue"
... so the nice thing about the about the cloud is that you can workaround cloud-specific issues?
Is it wrong to check DNS first? No, but please realize that DNS misconfiguration is not more common than other SNAFUS.
It’s not DNS
There’s no way it’s DNS
It was DNSUnfortunately, no one has the time to do that (or let someone do it) after the problem is "solved", so over time the "rebuild from scratch" approach just results in a loss of actual troubleshooting skills and acquired knowledge --- the software equivalent of a "parts swapper" in the physical world.
I don't really fault the junior dev who does that. They are just following the "I don't understand something, so just start over" paradigm set by seniors.
A special case of this is git rebase after which you "can" access the original history in some obscure way until it's garbage-collected; or you could clone the repo before the merge and then you can access the original history straightforwardly and you decide when to garbage-collect it by deleting that repo.
I would never ridicule anyone for your approach, just gently encourage them to spend a few mins to grok the ‘git reflog’ command.
- Commit any pending changes.
- Make a git tag at your current head (any name is fine, even gibberish).
If anything “goes wrong” you can rollback by simply doing reset hard to the tagged commit.
Once done, delete the tag.
Making a complete “backup clone” is a complete waste of time and disk space.
The two primary source code management activities developers use are versioning of source code (tracking changes which happened over time) and the other being synchronisation of code with other developers.
One of Git’s differentiating strengths is it being decentralised, allowing you to do many operations in isolation locally without a central server being involved. You can then synchronise your local repository with an arbitrary number of other copies of it which may be remote, but you may need to rebase or merge in order to integrate your changes with those of other developers.
Git is more like a local database (it even allows multiple local checkouts against a single common “database”) and it only occasionally “deletes” old “garbage”. Anything you do locally in Git is atomic and can always be rolled back (provided garbage collection hasn’t yet been performed).
Although I’m comfortable enough with using the reflog to rollback changes (I’m also skilled enough in git I haven’t needed to in many years), it’s not very user friendly, it’s essentially like sifting through trash, you’ll eventually be able to find what you lost (provided it wasn’t lost too long ago), but you may have to dig around a bit. Hence my suggestion of tagging first, makes it easy to find it again if needed.
I have very limited Linux experience and have no recommendations on your other question.
If you have proper observability in place then you can do your diagnosis without affecting your customers.
Plus, at the same time successful diagnosis is also the kind that can have the most dramatic effect on your customers.
In a positive way.
I get the desire to troubleshoot but priority 0 is make the system functional for users again, literally everything else can wait. I once had to deal with an outage that required we kill all our app servers every 20 minutes (staggered of course) because of a memory leak while it was being investigated.
What numbers went into this calculation, to get such an extreme result as concluding that getting it up again is always the first priority?
When I tried to estimate the cost and benefit, I have been surprised to make the opposite conclusion multiple times. We ended up essentially in the situation of "Yeah, sure, you can reproduce the outage in production. Learn as much as you possibly can and restore service after an hour."
This is in fact the reason I prefer to keep some margin in the SLO budget -- it makes it easier to allow troubleshooting an outage in the hot path, and it frontloads some of that difficult decision.
But consider that the case for older companies, where it was more common to deploy several systems, often complex ones, onto the same node. You will also cause outages to system x, y and z too. Maybe some of them are inter-dependent? You have to outwhey the consequences and risks carefully in any situation before rebooting.
At least as I read it, this contains the assumption that that‘s not how you deploy your applications
Yeah we do this? It doesn’t pose an issue though. Cordon the node (stop any new deployment going on), drain it to remove all current workloads (these either have replicas, or can be moved to another node, if we don’t have a suitable node, K8s spins up one automatically) and then remove the node. Most workloads either have replicas spare, or in the case of “singleton” workloads, have configs ensuring the cluster must always have 1 replica available, so it’s waits for the new one to come up before killing the old. Most machines deploy and join the cluster in a couple of minutes, and most of our containers take only like, 1 or 2 seconds to deploy and start serving on a machine, so rolling a node is a really low impact process.
So... more generally, your idea with replacing VMs is rather naive when it comes to storage. Replacement incurs penalties, s.a. eg. RAID rebuilds. RAIDs don't have the promised resiliency during rebuild. And, in general, rebuilds are costly because they move a lot of data / wear the hardware by a lot. Worst yet, if you experience the same problem that caused you to start a rebuild in the first place during the rebuild, the whole system is a write-off.
In other words, it's a bad idea to fix problems without diagnosing them first if you want your system to be reliable. In extreme cases, this may start a domino effect, where the replacement will compound the problem, and, if running on rented hardware, may also be very financially damaging: there were stories about systems not coping with load-balancing and spawning more and more servers to try and mitigate the problem, where problem was, eg. a configuration that was copied to the newly spawned servers.
You can probably see where this is going. The workers had some problem where they would bog down if left running too long. Causing the queues to back up and indirectly causing themselves to eventually be culled.
Never did figure out why they would bog down. We just ran herky jerky like this for a few years till I left. Might still be doing it for all I know.
So you just automatically replace the instances after a certain amount of runtime and your problem is gone.
Is this really a prevailing attitude now? Who cares what happened, as long as we can paper over it with some other maneuver/resources? For me it's both intellectually rewarding and skill-building to figure out what caused the problem in the first place.
I mean, I hear plenty of managers with this attitude. But I really expect better on a forum called hacker news.
If it happens extremely rarely (like, once every 6 months) or it’s super transient and low impact, we kick it and move on.
If it starts happening a 3rd or 4th time, or the severity increases we start to dig in and actually fix it.
So we’re not giving up, and losing all diagnosis/bugfixing ability, just setting a threshold. There’ll always be issues, some of them will always be mystery issues, so you can’t solve everything, so you’ve got to triage appropriately.
Unfortunately, many such tools in docker images will be flagged by automated security scanning tools in the "unnecessary tools that can aid an attacker in observing and modifying system behavior" category. Some of those ( like having gdb) are valid concerns but many are not.
To avoid that we have some of these tools in a separate volume as (preferably) static binaries or compile & install them with the mount path as the install prefix (for config files & libs). If there's need to debug, we ask operations to mount the volume temporarily as read-only.
Another challenge is if there's a debug tool that requires enabling a certain kernel feature, there are often questions/concerns about how that affects other containers running on the same host.
Starting a second container is usually a good idea anyway, since you need to add a lot of extra flags like SYS_PTRACE capability, user 0 and --privileged for debuggers to work.
This way you don't need to restart the prod-container either, potentially loosing reproduction-evidence.
Remembering how to do all this in an emergency may not be entirely obvious. Make sure to try it first and write down the steps in your run books.
That was our initial idea. But management and QA are paranoid enough that they consider these as new set of images that require running the complete test suite again even when they are built on top of certified images. Nobody is willing to test twice, so we had to settle for this middle.
https://man.freebsd.org/cgi/man.cgi?rescue https://github.com/freebsd/freebsd-src/blob/main/rescue/resc...
These were a lifesaver multiple times.
I've recommended atop in the past for catching short-lived processes because it uses process accounting, although the newer bpf tools provide more detail.
I didn't, but a boss or two of mine did.
Build a container with a one-liner:
docker build -t tcpdump - <<EOF \nFROM ubuntu \nRUN apt-get update && apt-get install -y tcpdump \nCMD tcpdump -i eth0 \nEOF
Run attached to the host network:
docker run -dP --net=host moremagic/docker-netstat
Run system tools attached to read host processes:
for sysstat_tool in iostat sar vmstat mpstat pidstat; do alias "sysstat-${sysstat_tool}=docker run --rm -it -v /proc:/proc --privileged --net host --pid host ghcr.io/krishjainx/sysstat-docker:main /usr/bin/${sysstat_tool}" done unset -v sysstat_tool
Sure, yum install is preferred, but so long as docker is available this is a viable alternative if you can manage the extra mapping needed. It probably wouldn’t work with a rootless/podman setup.
apt libs borked or something?
Of course handling dependencies by hand is annoying, but depending on situation it might be faster anyway.
There are groups of people you always make nice with.
* Security people. The kinds with poorly fit blazers who let you into the building. Learn these peoples names, Starbucks cards are your friends.
* Cleaning people. Be nice, be polite, again learn names. Your area will be spoltless. It's worth staying late every now and again just to get to know these folks.
* Accounting: Make some friends here. Get coffee, go to lunch, talk to them about non work shit, ask about their job, show interest. If you pick the right ones they are gonna grab you when layoffs are coming or corp money is flowing (hit your boss up for extra money times).
* IT. The folks who hand out laptops, manage email. Be nice to these people. Watch how quickly they rip bullshit off your computer or wave some security nonsense. Be first in line for every upgrade possible.
* Sysadmins. These are the most important ones. Not just because "root" but because a good SA knows how to code but never says it out loud. A good sysadmin will tell you what dark corners have the bodies and if it's just a closet or a whole fucking cemetery. If you learn to build into their platform (hint for them containers are how they isolate your shitty software in most cases) then you're going to get a LOT more leeway. This is the one group of people who will ask you for favors and you should do them.
like, how? are you straight up bribing people with coffee for security favors? or is it like, "hey man, thanks for helping me out I'd like to buy you a coffee but I'm busy with secret consulting stuff - here's a gift card"
Is this something that only works for short lived external consultant interactions?
By the end of a 4-5 week run you will know all the security people in a building. If I go to lunch an forget my badge they will let me back in no questions asked. This is something I used to do as staff, and still do to this day.
Its not about bribing to get a specific favor. Its about getting on good terms. Having people like you is a good thing and it makes their job a bit better and can make their day a little brighter. win-win
Network connectivity issues aren't always apparent in some apps.
Used all of these and more, in Unix, not just Linux crisis situations.
Most dev teams I work with are actively reducing their actual managed server and replace it with either Lambda, or docker images running in K8. I wonder if these tools are still useful for containers and serverless?
There are plenty of services that don’t fit on k8s or Lambda. Not all pegs fit in those holes.
It's definitely harder for apps running in Lambda because we may not have access to the underlying OS. In such case, I kind of fallback to using the application level observability tools like Pyroscope (https://pyroscope.io). It doesn't always work for all the cases and have some overheads/set up but it's still better than flying and more useful than the Cloud Provider's provided metrics.
Usually not even a shell is available in modern Kubernetes deployments that take a security first approach, with chiseled containers.
And by creating a debugging image, not only is the execution environment being changed, deploying it might require disabling security policies doing image scans.
I imagine these problems described happen on big iron type hardware clusters that are extremely expensive and spare capacity isn’t possible. I might be wrong but especially with (sigh) AI setups with extremely expensive $30k GPUs and crazy bandwidth between planes you buy from IBM for crazy prices (hardware vendor on the line so quickly was a hint) you’re way past the commodity server cloud model. I have no idea what could go wrong with such equipment where nearly ever piece of hardware is close to custom built but I’m glad I don’t have to deal with that. The debugging on those things work hardware only a few huge pharma or research companies use has to come down to really strange things.
Then you have storage, often tiered in three levels - job-temporary scratch storage on each node, a distributed fast storage with a few weeks retention only, and an external permanent storage attached somehow. Relatively often the middle layer here, which is Lustre or something similar, can throw a fit.
Then you have the interconnect, which can be anything from super flakey to rock solid. I've seen fifteen year old setups be rock solid, and in one extreme example a brand new system that was so unstable, all the IB cards were shipped back to Mellanox and replaced under warranty with a previous generation model. This type of thing usually follows something like a Weibull distribution, where wrinkles are ironed out over time and the IB drivers become more robust for a particular HW model.
Then you have the general hardware and drivers on each node. Typically there is extensive performance testing to establish the best compiler flags etc., as well as how to distribute the work most optimally for a given workload. Failures on this level are easier in the sense that it typically just affects a couple of nodes which you can take offline and fix while the rest keep running.
This probably would have prevented a stressful incident 3 weeks ago.
The times I've been locked in an ill-equipped container image that was stripped bare by some "security" crapware and/or guidelines and that made debugging a problem MUCH harder than it should have been vastly outnumber the times where I've had to deal with a security incident because someone had coreutils (or w/e) "unnecessarily" installed. (The latter tally is at zero, for the record.)
(This is a genuine question). In what circumstances would you need to install/run dev tools in prod?
Of course having a compiler installed isn't necessarily an issue... but it might well be a sign that there is an underlying problem!
(FWIW, I used to build everything from source. Yes, also in prod. That was a while ago...)
This is one of the reasons why I fight back as hard as I can against any "security" measures that restrict what root can do.
[EDIT]: typo