Comment by chrissnell | Hacker News Reader

Comment by chrissnell | Hacker News Reader

The timing of this announcement stinks. 2130 PDT on a Friday night, long after most folks have gone home. Making it even more painful, Rackspace is providing 1 hour advance notification. For those of us hosted in the US, there's a rolling reboot window that starts at 0400 PDT on Sunday morning. So, if you're a Rackspace customer and care that your app shuts down cleanly and restarts properly, you get to wake up at 0400 PDT and check your email and stay near your laptop and internet at least once an hour, every hour, until (potentially) 0400 PDT on Monday morning. Hooray, my Sunday plans are fucked!

They suggest taking backups/snapshots of your instances before the reboot window. Given the throughput required to push multi-hundred GB images from public cloud servers to Cloud Files for storage, I am willing to bet that the backup network is maxed out and will stay maxed out until the outage.

I wonder if Rackspace found out about the rumored Xen exploit from the same people that told Amazon, or if Amazon told Rackspace but waited a little bit to make it more painful for Rackspace's customers...

The timing of this and the wording of their announcement leads me to think that they weren't included in the select group of people who are brought up to speed with the Xen security update before it's released.

The next question then is, who does have access to this? It definitely needs to be kept wrapped-up at least until the large vendors are patched, but who gets to decide who's a large vendor?

The Xen pre-disclosure list is public, and Rackspace is on it:

http://www.xenproject.org/security-policy.html

They also spell out the policy for who is eligible.

Thanks for the link. Given that Amazon was notifying customers as early as two days ago, we can assume that Rackspace received notification from Xen at least two days ago. They waited until Friday night to announce this? Awful.

prgmr.com also got inducted 2 days ago http://blog.prgmr.com/xenophilia/2014/09/prgmrcom-now-on-xen...

Xen has previously had problems with people leaking embargoed security issues http://www.gossamer-threads.com/lists/xen/devel/248239

Any word on if/when Linode may schedule reboots?

https://forum.linode.com/viewtopic.php?p=64914#p64914

Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy. It genuinely sounds like you don't trust it to come back up which is not something I could sleep on. Things fail, sometimes violently.

Even with our company which has in in-house ops team and our own colocated kit, we expect process failures and restarts and plan accordingly. No sleep is lost even when something explodes at 2AM (other than for the ops team, who in this case you contracted out).

You're missing his point, and it's condescending/naive of you to assume that his app hasn't been tested to withstand it with no information to tell you that. My app has been thoroughly tested to withstand reboots; however, I have 22 machines with rackspace and (a) I've never tested all 22 going down at random times, and (b) I STILL want to be awake in case anything happens unexpectedly. The pain here is the timing and announcement window for those of us who want to monitor their machines for problems.

Well when you contract out your infrastructure that really should be part of your DR plan. Analysing failure conditions is really priority one when you put your stuff on someone else's turf as you have little control over this, despite contracts etc. You did do a DR plan right and did check your contract with RackSpace?

I want to be woken up if something doesn't come back, not if it does or even if it has gone into limp mode.

Seriously you're looking at the wrong end of the problem.

Jesus christ dude, we get it, you're a perfect sysadmin who has clearly covered all possible bases.

But others are not like you. Systems are not always 100% foolproof, people don't have comprehensive DR plans.

Is it that hard for you to understand why someone may want to be up when there is a reboot? Like seriously?

> Sorry but in this case your application should have been tested to withstand all possible failure scenarios and restart scenarios so I have little sympathy.

Someone who thinks they can test all possible failure scenarios is severely lacking in imagination.

Sorry I should have used the word probable which is limited to power, network, software and storage failures. The solution to these is redundancy and to test each failure condition in succession and combinations thereof.

Sure there are more possibilities, but these should all be covered with a complete fallback DR strategy i.e. if not identifiable cause X, Y or Z then assume the worse, snapshot everything for debugging later and fallback to a complete restore from scratch.

The trick is to make X, Y and Z self-healing and part of your architecture provisioning.

I get the feeling most people here haven't dealt with large deployments and multiple site redundancy, five-nines reliability requirements and extensive DR planning, because the above is pretty obvious to those of us who have.

Well it appears that 20 years' experience doesn't trump some total amateurs with a cloud to play with...

Reply on news.ycombinator.com