Hacker News Reader

Comment by aloknnikhil | Hacker News Reader

> I asked for an API to do a hard power off of an instance. To this day, you can only do a graceful power off. I want to know what happens when the instance just goes away.

Wouldn't just running "halt -f" do the same?

View on HN

Possibly, but how can we be 100% sure without the ability to compare behavior? If I were following this line of research I'd still want to know if there's any difference in the nature of a failure when it comes from within the OS (possibly simulated by half -f) and the situation the parent OP pointed out where the instance just goes poof without sending any kind of signal to the OS itself.

Though of course then the trouble is, if AWS is simulating specific behaviour, it won't be exactly the same as real problems when they occur. It is a bit better, but hard to say how much better.

I'd think the key on this is being able to simulate very specific partial-failure conditions. e.g. specific packet loss, loss of connections to EBS, etc. Just turning machines off I expect wouldn't be that valuable.

> Though of course then the trouble is, if AWS is simulating specific behaviour, it won't be exactly the same as real problems when they occur.

If Amazon simulated power failure, or network cable disconnect, or potentially even corrupt writes to disk, I personally feel it would be indistinguishable from the event really happening.

The problem is that they can't really realistically simulate that for a single instance unless you've got the whole physical machine to yourself, and of course the worst issues tend to be when it's half-broken, not entirely off/disconnected

Power loss or network disconnect could easily be simulated without needing the whole physical machine.

Kernel panics work very well for this use case.

echo c > /proc/sysrq-trigger

For reference: https://unix.stackexchange.com/a/66205 (you obviously need to be root)}

Maybe for testing the crash-resilience of software running on the node; but not necessarily for testing how the SDN autoscaling glop you've got configured responds to the node's death.

A panicking instance is still "alive" from its hypervisor's perspective (either it'll hang, sleep, or reboot, but it won't usually turn off its vCPU in a way the hypervisor would register as "the instance is now off"); while if a hypervisor box suffers a power cut, the rest of the compute cluster knows that the instances on that node are now very certainly off.

echo o > /proc/sysrq-trigger will shutdown a AWS instance immediately. I've used it a couple of times to make real sure my costly spot instance was terminated.

https://www.kernel.org/doc/html/v4.11/admin-guide/sysrq.html...

https://elixir.bootlin.com/linux/latest/source/kernel/power/...

Looks like SysRq-o does a clean poweroff, not a dirty/immediate one — it calls kernel_power_off(), not machine_power_off().

This means, importantly, that can actually take some time to happen, as drivers get a chance to run deinitialization code. This also means that a wedged driver that doesn't respond during deinit can prevent the kernel from halting.

Thus, while SysRq-o might be useful for killing a wedged userland, it's not a panacea — especially, it isn't guaranteed to complete a shutdown for unstable kernels, or kernels with badly-written DKMS drivers attached. It's not truly equivalent to a power-cut.

To give an example of the difference: does it fsync() before stopping?

It would be really close yes, but not exactly the same as ripping the power cord out. It still gives the OS a hint that shutdown is coming.

Not exactly. "halt -f" depends on some variant of power management working. For extra fun, you could also have partial power-offs, like say a power cut to the motherboard, or any combination of other devices with their own power lines from a power supply. (in a current PC these would typically be PCI supplementary power, motherboard, SATA power, 4-pin 12v molex power, supplemental CPU power.)

Granted, most of those are out of scope for cloud development, but the concept of externally cutting off a VM is different than even calling whatever power service you have to cut power. In the 'real' world, enough of those failure triggers above would probably also trigger an automated power cycle if you're in a managed environment.

To elaborate a bit further - when things are fine, they're, well, fine. Unfortunately, the assumptions of how computers actually function go out of the window when enough things go wrong, like lightning striking the wrong thing. Things go weird sometimes, like the accepting TCP connections, but not sending responses. can't ping but can SSH in, or vice versa. In a cluster with automatic failover, you want to be sure the bad node is dead dead dead because a bad machine can cause harm to all the packets in the rack. (Eg HA with floating IPs and ARP takeover.) Thus the requirement is to forcefully disable the bad node, which is where the acronym STONITH comes from - Shoot The Other Node In The Head. (HA clusters with more than two nodes existed, but were rarer than 2-node clusters simply due to cost.) Before cloud computing was so pervasive, high availability was implemented with serial cables between in physical hardware and when one machine stopped responding, the equipment basically yanked its power cable out.

Reply on news.ycombinator.com