When power cycling your (x86) server isn't enough to recover it

In grad school we built a lot of logic boards from scratch. They were used for submersible robots, and we had a 350,000 gallon water tank that we kept heated to 88 degrees. This was three stories above the ground in a metal building. You can't really air condition that, so in the summer it got quite hot.

It was not uncommon to return from lunch to find than an embedded computer board that had been working when you left wasn't any more. One way to debug them was to put them in the refrigerator for a while. If they then worked, you knew you had a bad solder joint or an IC that was on the verge of failing.

I have at least 2 regular cases where full power off was required to resolve the issue.

First one is dell latitude laptop with fingerprint reader, randomly after few days of operation, fingerprint reader stops responding and login screens freeze for a minute until it timeouts few times. reboot does not solve it, nor suspending machine. it needs to be powered off and on again (hibernation to disk also works).

second case is my pc with ASRock creator x570, after long time if keeping it suspended, WiFi card stopped to function and just throwed some errors in dmesg on driver initialization. here even power off and on did not help, but flipping switch on power supply for few second resolved the issue

A BIOS can forget to reset some devices. A physical device might have a design flaw where it forgets to reset some registers on reset. A BIOS (including device firmware) can forget to zero some RAM/initialise a structure and get lucky.

Are these Dells?

Some Dells have a "feature" where something, somewhere, in their mess of a UEFI/iDRAC stack will get corrupted and will stay wrong through power cycles until you physically unplug the servers from power and hold down the power button to discharge a capacitor and clear out the NVRAM where the corrupted value is.

Most recently this impacted a PowerEdge R7525 server we have where the iDRAC was enforcing a power cap of ~300 watts leaving the system to be less than 1/10th as performant as it should have been. Manually setting a new power cap did nothing except update the values displayed in the UI. Multiple six minute (because of their mess of a UEFI/iDRAC stack) reboots of both the server and the iDRAC did nothing.

Dell was less than useful except for the fact that they hosted the answer. After raging against their CSA script/LLM auto-reply bullshit for days an aggrieved user with the same issue looking for help in their forums finally posted that he did the cap drain trick and it worked.

Saved me tons of wasted time. Thanks, anonymous fellow frustrated dell customer!

The Linux kernel supports rebooting using a number of different strategies[1]. Some PCs need a different one than the default in order to make sure everything is properly reset.

[1]: https://github.com/torvalds/linux/blob/9b2ffa6148b1e4468d08f...

Whenever I power cycle something that doesn't go right the first time, I leave it off for at least 30 seconds so all the caps can discharge and any saved state can reset. Especially true of routers etc.

I ran MECleaner once, and removed power from a desktop, waited ten seconds, plugged it back in, and the test for the presence of the ME was still positive.

I unplugged it and left it overnight, and the next day, the ME was gone.

This was the ARC version, but it can remain operational for some time after power is removed.

I usually prefer the 'reset' option (such as in IPMI). After all, this is the as-designed way to politely ask all devices to re-initialize.

Yes, power-cycling is more unambiguous, but afaikt, the example here is purely that power cycling really needs a noticable off-period so that all devices can fully come down. Otherwise, there's no real standard on what should happen - this or that component might stay up or retain state.

The other reason I like 'reset' is that lots of devices (fans, disks, probably all power systems - definitely including PSUs) have lifetime limits in power cycles. Mostly this is minor, unless you do something like reboot cluster nodes after a job (concievably a paranoid security requirement), or some automation gets in a loop and continually zaps a server.

OP, what Linux is this? I'm really curious, I don't recognize that trace format and I can't find the code to print exception traces with the eight bangs on the first line like that anywhere in the upstream git history. I think they're actually from the BIOS?

   !!!! X64 Exception Type - 12(#MC - Machine-Check)  CPU Apic ID - 00000000 !!!!

My story: I had an Intel NUC running Linux back in the day, which would get stuck in standby such that I had to remove and replace the CMOS battery to get it to boot again! I never figured that one out...

I've seen similar behavior when trying out a fork bomb in the terminal on both Linux and Windows. My guess is that on windows the fork bomb made it into the virtual memory and was recorded to disk and wasn't cleaned out completely during boot.

It too, 3 reboots to clear up the errors. Generally on the linux system one extra reboot was necessary about half of the time.

I’ve came across Acer laptops that’d always bluescreen on restart after a PROCHOT shutdown. The fix is to pull out the battery for a few seconds and then plug it back in, magically fixes the bluescreen.

This sounds like something the younger generation is having to relearn. When you power down a machine, leave it off for at least 30 seconds. One to let the various capacitors drain & discharge, and Two to let any network devices see that the machine is off so they update their internal tables.

You want your cold boot to truly be a cold boot.

I think it was the 1970's when I first heard of the "remove power, wait a good while, try again" strategy.

The subject was a cheap little black & white TV set that my folks had. Dad was an amateur radio operator, who mostly built his own equipment. He could have dissembled it, traced circuits, and calculated the wait time if he'd cared to.

Something like this happened to me once. Lost power in a lightning storm and when it came back my computer could no longer shut off.

Like, at all. Would just hang when you tried. Couldn’t exit from BIOS after changing settings, couldn’t suspend to RAM. Had to yoink the cord whenever I needed to restart. Wild stuff.

Perhaps like Frankenstein the lightning was a breath of life, and with its new sentience my PC was trying to preserve its existence. At any rate I reflashed the BIOS after a few months and it never happened again.

When managing large numbers of Dell rack mounted servers, a flea power drain is something you become very familiar with.

I had a desktop that would so something similar occasionally ~15 years ago. I am impatient, so rather than leaving it off for a while I would unplug, hit the power button, plug it back in and turn it on. Usually the fans would even spin for a fraction of a second, there was so much residual power in the caps.

I recently discovered my motherboard has a quirky Bluetooth chip which sometimes deadlocks and a simple power cycle was not enough to restore. The machine actually had be power cycled for more than a minute (!). I was stumped until I found the one reddit comment suggesting I try it... and sure enough.

This sort of issue is a relatively regular occurrence in the server fleet I and my team deal with. A handful of servers a month (out of thousands) end up misbehaving - a system won’t respond to a power up command from the BMC, a PCI-E device doesn’t appear properly after boot up etc etc

Standard troubleshooting before getting the vendor involved or replacing parts is to action a power drain.

BMCs with high uptime can be especially prone to this, often forgetting how to talk to the system they’re attached to.

I've experienced a similar problem with a Thunderbolt port on a machine. Nothing that plugs into the machine would be recognized. Not even a simple USB device. Power cycling multiple times didn't fix it. But powering off and leaving the machine off for a few minutes fixed it.

Given the problem occurred only once, I didn't do any more investigation on why.

I have had several laptops over the years like this. Full shutdown and power on does not reset some problems, like missing audio, missing wifi, etc. For Lenovo devices, I have to go as far as using the 'recovery' button. This goes for DP Alt Mode as well. Kinda annoying, but at least there's a solution.

I don't get it. That post was a whole bag of nothing. Why are you guys upvoting it?

I had the problem on APU4C4, iirc. You install openwrt on it, everything is working fine, then, you reboot and you get nothing on the serial port.

You unplug/plug it, cold boot it, and then it works again.