Cloud Server Reboots | Hacker News Reader

Cloud Server Reboots

status.rackspace.com

68 pointszorpner39 comments

Those of you who are good at sysadminning don't need this advice, but for the fellow people who are only borderline competent in the room, pay particular attention to this reboot if you recently did "apt-get update" or similar to take care of the bash problem. I've shot myself in the foot before and accepted new updates to e.g. mysql that caused the existing config file to raise a hard error on load, which was only discovered the next time the server was restarted. As you can imagine, inability to boot the database has unpleasant consequences for web apps connected to it.

I've never been a "real" sysadmin, but back in the mid 80s I learned that anything you don't practice doing regularly will inevitably be painful.

I was working as a developer at Unisys (or it might have still been Burroughs), down the hall from the OS developers (for one of Burroughs' 3 distinct mainframe product lines, each with its own cpu architecture, operating system, system software, etc.). At that time, our mainframes could be patched while running, without re-booting; re-booting was considerably slower than patching. As a result, as each new OS patch was developed, it was applied on the fly to the machines in our dev environment, and they hadn't been rebooted for something like six months. It turned out that it was possible to have a sequence of patches, which when applied on the fly to a running system caused no problems, except that when it came time to reboot, it would crash. You can imagine the fun of trying to debug which change (or combination of several changes), out of many months of development work, had broken the boot process! These dev machines were shared with other teams (compiler developers, database engine developers, etc.), so the productivity hit waiting for our dev machines to be available again was substantial. After that, the policy was that the OS developers took turns coming in early one morning a week to reboot the dev machines, so in the worst case they'd only have to track down which of the last week's worth of patches might have broken the boot process.

Sounds like it'd be a good idea to perform a deliberate reboot during off-peak hours and preemptively fix any issues.

You would think so (and I won't say it isn't), but if you're just one guy handling a few servers for your hosted SaaS, it's easy to fall into the "don't disturb the dust" trap -- especially once you've had a routine maintenance reboot or upgrade completely bone you.

Not too long ago I had a 36-hour straight marathon sysadmin session when a routine update happened to break my particular stack without advance notice on a web server used by a bunch of customers. I think my eye still twitches when I think of it.

Usually the next thing people say is, "you should set up x, y, and z tools for better redundancy / failover / load distribution / backups / sysadmin management / etc." Again, they're not wrong, but none of those things directly makes you more money, and making time for them can be difficult, especially since so few of them can be set up as easily as they claim on the box.

Don't disturb the dust is actually a fairly good heuristic.

It sure would have helped those guys working at Chernobyl.

Rather than doing that, manage your server(s) with code. While there is a little overhead in learning something like ansible, being able to provision another machine to do a dry-run more than makes up for that cost.

    pay particular attention to this reboot if you recently
    did "apt-get update" or similar to take care of the bash
    problem

I don't understand. apt-get update won't install new software on reboot. That just updates the index in apt. There's no other way to update bash without "apt-get update && apt-get install bash" and that won't impact anything on reboot.

Your updated mysql config file sounds like you apt-get upgrade'd, installed a new version of mysql, or something like that and the changes didn't go into effect until the mysql process was restarted. I got bit in the ass with that same thing when I apt-get upgrade'd.

Good thing I've I never totally hosed myself when picking the wrong word/letter in a command line as opposed to a non-executable HN comment.

cries softly to self

Two things like this; my tomcat config got reset to max 128mb JVM stack and ubuntu seems to have switched from -virtual to -generic kernels but the latter isn't booting for me.

This isn't to handle the bash problem?

What is it to fix then?

nobody is saying yet, but probably the xen issue that's scheduled for disclosure on october 1

There is a bug in Xen, which hasn't been publically annouced yet.

The timing of this announcement stinks. 2130 PDT on a Friday night, long after most folks have gone home. Making it even more painful, Rackspace is providing 1 hour advance notification. For those of us hosted in the US, there's a rolling reboot window that starts at 0400 PDT on Sunday morning. So, if you're a Rackspace customer and care that your app shuts down cleanly and restarts properly, you get to wake up at 0400 PDT and check your email and stay near your laptop and internet at least once an hour, every hour, until (potentially) 0400 PDT on Monday morning. Hooray, my Sunday plans are fucked!

They suggest taking backups/snapshots of your instances before the reboot window. Given the throughput required to push multi-hundred GB images from public cloud servers to Cloud Files for storage, I am willing to bet that the backup network is maxed out and will stay maxed out until the outage.

I wonder if Rackspace found out about the rumored Xen exploit from the same people that told Amazon, or if Amazon told Rackspace but waited a little bit to make it more painful for Rackspace's customers...

The timing of this and the wording of their announcement leads me to think that they weren't included in the select group of people who are brought up to speed with the Xen security update before it's released.

The next question then is, who does have access to this? It definitely needs to be kept wrapped-up at least until the large vendors are patched, but who gets to decide who's a large vendor?

The Xen pre-disclosure list is public, and Rackspace is on it:

http://www.xenproject.org/security-policy.html

They also spell out the policy for who is eligible.

Thanks for the link. Given that Amazon was notifying customers as early as two days ago, we can assume that Rackspace received notification from Xen at least two days ago. They waited until Friday night to announce this? Awful.

prgmr.com also got inducted 2 days ago http://blog.prgmr.com/xenophilia/2014/09/prgmrcom-now-on-xen...

Xen has previously had problems with people leaking embargoed security issues http://www.gossamer-threads.com/lists/xen/devel/248239

Any word on if/when Linode may schedule reboots?

https://forum.linode.com/viewtopic.php?p=64914#p64914

Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. It genuinely sounds like you don't trust it to come back up which is not something I could sleep on. Things fail, sometimes violently.

Even with our company which has in in-house ops team and our own colocated kit, we expect process failures and restarts and plan accordingly. No sleep is lost even when something explodes at 2AM (other than for the ops team, who in this case you contracted out).

You're missing his point, and it's condescending/naive of you to assume that his app hasn't been tested to withstand it with no information to tell you that. My app has been thoroughly tested to withstand reboots; however, I have 22 machines with rackspace and (a) I've never tested all 22 going down at random times, and (b) I STILL want to be awake in case anything happens unexpectedly. The pain here is the timing and announcement window for those of us who want to monitor their machines for problems.

Well when you contract out your infrastructure that really should be part of your DR plan. Analysing failure conditions is really priority one when you put your stuff on someone else's turf as you have little control over this, despite contracts etc. You did do a DR plan right and did check your contract with RackSpace?

I want to be woken up if something doesn't come back, not if it does or even if it has gone into limp mode.

Seriously you're looking at the wrong end of the problem.

Jesus christ dude, we get it, you're a perfect sysadmin who has clearly covered all possible bases.

But others are not like you. Systems are not always 100% foolproof, people don't have comprehensive DR plans.

Is it that hard for you to understand why someone may want to be up when there is a reboot? Like seriously?

> Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy.

Someone who thinks they can test all possible failure scenarios is severely lacking in imagination.

Sorry I should have used the word probable which is limited to power, network, software and storage failures. The solution to these is redundancy and to test each failure condition in succession and combinations thereof.

Sure there are more possibilities, but these should all be covered with a complete fallback DR strategy i.e. if not identifiable cause X, Y or Z then assume the worse, snapshot everything for debugging later and fallback to a complete restore from scratch.

The trick is to make X, Y and Z self-healing and part of your architecture provisioning.

I get the feeling most people here haven't dealt with large deployments and multiple site redundancy, five-nines reliability requirements and extensive DR planning, because the above is pretty obvious to those of us who have.

Well it appears that 20 years' experience doesn't trump some total amateurs with a cloud to play with...

And this is why live migrations (VMware vMotion, but also done by Google Compute Engine) are so awesome. Migrate VMs from server X to server Y, then patch and reboot server X. No VM downtime.

Yeah, Xen supports live migration as well (at least, libvirt supports live migration of xen backends: http://libvirt.org/migration.html). I would be interested in hearing why the big providers aren't apparently using it.

I envy the AWS users who enjoyed the rolling reboots (which were AZ aware!) across a small minority of the EC2 fleet. (~10%, yeah?)

At some point on Sunday, I'm going to be picking the pieces of our entire stack.

Rackspace doesn't even offer anything like availability zones.

The last major maintenance they scheduled was over the July 4th weekend -- wasn't happy with that one either.

Re: availability zones; while technically true, I'm not sure that's a fair comparison. Rackspace provides uptime guarantees for the internal network per monthly billing period, and they do actually organize the DC in cells (and yes, they do the roll-outs cell-aware). Those cells are simply not end-user visible (AFAIK).

For this particular maintenance, we're being told that Rackspace will be proceeding with all cells in a region, in parallel. They will not be respecting cell boundaries.

if the "cells" aren't visible in such a way that you can distribute your application across them, it doesn't really matter that they exist.

How so? It seems like a sufficiently smart comp^H^H^H^Hservice could figure out how to distribute machines across them, even if the user doesn't know they exist.

Turns out it's a moot point anyway, because cells are client-visible, and have been contributed upstream to OpenStack as a Nova extension.

It's not a moot point. Just because cell info is in Nova doesn't mean it's publicly visible.

The only way we find out about cell locality is when our account rep gives us an updated spreadsheet. We've inquired about this several times.

If you happen to actually use the Rackspace public cloud and know a specific API call we're missing, I'd love to hear it.

the Xen vulnerability must be something severe if they are all doing this. [1] http://xenbits.xen.org/xsa/

just imagine what all those ISPs and hosts who are not on the private list but who use Xen are thinking. They must be crapping themselves, hitting F5 on the announcements page, ready to pounce and fix.

With AWS and shell shock, it's an interesting time to have just started working for a managed hosting company, nothing like being thrown straight into the deep end of being on call!

Well that is a service I won't use then.

First of all you communicate all details so I know how I will be affected. Secondly you don't shut down my service ever, for any reason, other than lack of payment.

If you can't do those things you don't get to claim to have excellent support.

I don't think you understand the gravity of the situation. It's a choice between rebooting Xen hosts or having them (and you) get owned when the exploit is made public.

Reply on news.ycombinator.com