Hacker News Reader

Comment by raffraffraff | Hacker News Reader

Have you tried turning it off and then on again?

> We're no longer pursuing restarts as a path to remediation.

Oh you have

Kind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it.

> We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue.

At least it didn't fix the problem so they can actually start finding the real cause.

> We're no longer pursuing restarts as a path to remediation.

Why isn't the AI they sell telling them what's wrong? Why do they need to take shots in the dark to "see if that resolves the issue"?

"Yeah, I'm with Rob. Just let's reboot and see what happens"

"If that doesn't work, clear the cache and reboot again."

In my experience it’s a safe way to do something useful while everyone is getting their bearings. It immediately partitions the situation space between being persisted or systemic vs local or caused by long-running processes. Plus everyone’s going to ask if you’ve tried that already, so you might as well get it out of the way if it makes any amount of sense

Followed quickly by "Redeploying with more log lines", the next logical step.

I don't know, that reads exactly like an AI troubleshooter working through a plan without the implicit contextual understanding an experienced human might bring to either the actions or the communications.

"Oops, we forgot to tell it that this is the hyperscaled Salesforce production environment and that its choices need to project competence and consider brand embarrassment. WILLFIX"

Restart should be a very last emergency step, as if it works, a restart often might wipe out evidence of why.

So hopefully it's not done often.

I wouldn't say so, rather you need to balance recovery time and evidence preservation. A good incident manager will give the service owning team a chance or two to debug, but not let them fall into the trap of needing to understand the problem fully before attempt a clumsy potential fix. And of course will take into account the total business impact of the ongoing disruption and the known and unknown risks of the proposed clumsy fix (it could make things worse).

Yes, that's why it's the last emergency step. We're not disagreeing.

Rebooting is the first step: If it fixes it you don't have a problem. If it doesn't, you know more about the problem.

It's a joke, but like, after nearly two decades of engineering I have something break on me, due to updates. I figure the updates broke it, call the vendor, and they go... did you try rebooting it again?

Rebooting it a second time fixed it.

The requirement is to get customers out of impact as #1 priority. If there's suspicions around memory/thread states a restart makes a lot of sense. Digging through logs and flight records takes a lot of time, customers are losing business in that time. If you're afraid to restart your service you need to work on your telemetry.

Been many an MIR that I've seen where after initial assessment, the next log entry was "service restart attempted"

In my experience this is very common on linux hosts. Not that you would do a full system reboot as a generic first attempt (this was much more common when I worked with Windows) but restarting a wedged or misbehaving service/daemon is a pretty common thing.

Have you tried turning it off and then on again?

Flashbacks to "Thank you for calling Three-Ten-DELL. Have you tried turning it of and turning it back on again?"

Reply on news.ycombinator.com