Hacker News Reader

Comment by walterbell | Hacker News Reader

Cloud providers typically have early access to security fixes and would be patched by now. Amazon has the ability to hot-patch Xen, a feature which is now being developed for general use.

This guest escape vulnerability (http://xenbits.xen.org/xsa/advisory-148.html) has been present since 2008 and was found by Alibaba, which recently joined the Xen advisory board, http://www.xenproject.org/about/in-the-news/192-xen-project-...

View on HN

In that case Amazon's wording would be pretty disingenuous - "AWS customers' data and instances are not affected by these issues" when the reality is more like "as far as we know nobody exploited this before we patched"?

Good point. In theory, they could have had other mitigations in place for this class of vulnerability, which they chose not to share with the GPL Xen community.

It depends on the version of Xen they are running. XSA-148 only affects Xen 3.4 and newer. If AWS is running an older version of Xen, they may never have been affected.

Since Amazon don't tell anyone about what they run, and don't release back to the broader Xen community, I guess we'll never know.

Right, "AWS customers' data and instances _ARE_ not affected" (emphasis mine) does not imply "AWS customers' data and instances _WERE_ not affected"

Breaches are a one-way state change, like death. If customer data WAS affected, then it IS affected, until customers take action such as disclosure, resetting passwords/keys, etc. If customer data was affected, AWS would need to advise customers to take remedial steps. Since they have not issued such advice, is it safe to assume that they had mitigation in place for this attack?

It could also be the case that they have monitoring systems on the host machines that would have reliably detected this issue being exploited, even before this specific vulnerability was discovered (or perhaps better to say, given the age of the bug, before being discovered by someone that disclosed it to Xen).

I just saw a bunch of our SoftLayer VMs get rebooted (with ahead notification) and they seem to use Xen. They probably got notification too.

Looks like my others (Digital Ocean, Vultr) use KVM (according to virt-what).

Did anyone see Amazon do reboots? I wonder if Amazon has some method of hot-patching kernel and Xen code under the hood? There are such things, and that would be some interesting secret sauce.

Earlier this year:

> Our team has been working around-the-clock to find ways to minimize the impact for those requiring a reboot. We're happy to share that we'll now be able to live-update ‎the vast majority of our older hardware for this Xen Security Advisory.

https://aws.amazon.com/premiumsupport/maintenance-2015-03/

There's been a few subsequent articles mentioning Amazon's ability to hot-patch Xen without a reboot. They didn't perform a reboot for this vulnerability.

Out of the 1,000+ instances I had running in AWS during that last major reboot -- there were only a handful of hosts affected. Sadly, some of them were the very primary core DBs -- but in general, it was still only a handful.

Luckily, many people also converted to HVM over PV already. IIRC it was largely PV instances in EC2 classic... but I can't be sure.

Memories are short. What about VENOM https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-3456 which affected both Xen HVM and KVM? Then there was http://xenbits.xen.org/xsa/advisory-110.html which is privilege escalation in HVM mode.

Regardless of the virtualization technology, you need to keep on top of disclosures. Any public provider running xen should be on the disclosure list by now http://www.xenproject.org/security-policy.html

Here's a talk and wiki page on Xen hot-patching (xsplice), from the Aug 2015 Xen Summit: http://wiki.xen.org/wiki/XSplice & https://youtu.be/C2R79Ob8KOA

Linode rebooted mine on 21st

without any serious infrastructure you can also hot patch single instructions where the bytes you want to change fall in an 8 byte alignment. though, probably this is not useful for this bug but I know other cloud providers do this for security patches.

Reply on news.ycombinator.com