This seems to be such an obvious feature that I'm surprised they are only adding this now.
I'm not an expert in hypervisors or anything like that and so I'm wondering what was stopping them from adding it in the past?
I'm not an expert in hypervisors or anything like that and so I'm wondering what was stopping them from adding it in the past?
Pretty much all hypervisors support serial consoles, but usually those interfaces are limited to trusted admins. For something like AWS, they'll also have to connect it from the hypervisor hosts into their public UI, and they can't trust the users.
If an instance wedges itself onto a state where I need console access, I'd just kill it and provision a replacement (ideally, my monitoring and automation will have done that already and not even have woken me up to tell me).
I'm not sure I'd be at all comfortable having irreplaceable single points of failure in AWS. (Though I do recognise that people use it that way all the time...)
Anything more esoteric than "a normal Ubuntu" can have a bug, e.g. it hangs with three network interfaces or similar.
What it is actually saying is not that it's a good approach to just throw away servers when they start having problems. What's important is having that ability when it is necessary.
Servers don't just randomly fail. If your instance goes down, sure, your automation will recover your system, but it will still be important to know why it failed, because it may be a symptom of a deeper issue.
It can be due to a hardware issue, but even when that happens, you don't throw away good hardware; you fix it and the server can return to full operation. In the cloud, though, you have no idea what the hardware is doing. Maybe the instance failed because the underlying host failed; maybe it didn't. You should still find out.
Even at not-very-high scale, AWS instances _do_ "just randomly fail", at least for all practical interpretations. I don't run anything like FAANG scale, only hundreds of instances rather than thousands or millions, and I see at least a few "random failures" a year (not including spot instances terminating, which I see in clumps every month or so).
I (almost) never try to repair broken a EC2 instance. Wherever I can, they'll be running totally stateless, and I just provision new ones and kill off old ones. I probably won't even bother investigating if it's a rare and singular problem on a known-reliable platform. If one instance wedges and gets replaced, I'll just have a note to investigate if it happens again any time soon. If we get a second failure, we'll go looking in logs and maybe keep and investigate the EBS volume.
For platforms running new-ish code, procedures are different. If we see dead instances after deployments we obviously investigate the new code/config there. But a fair chunk of clients where I am only get 6 or 12 (or even 24) month backend update cycles, if I've got dozens of instances running the same code for months on end and _one_ dies, we just bury it and replace it, and keep a closer eye on the rest of the "herd" for a week or two.
When you manage your own physical servers, you have more knowledge of your risk. The actual time of failure will still be random, but if you've been running a host for 5 years straight, you know the risk is growing.
But we're mostly agreeing here. In the scenario where you throw away a "randomly failed" instance, the historical stability is good evidence that it is due to a hardware failure, and you can just replace the instance and move on.
https://www.linode.com/docs/guides/using-the-linode-shell-li...
They've gotten so far without this functionality that I have to wonder what finally tipped the balance into their offering it.
s/vendor is shipping AMIs without sshd, and they/TLA/
A few people were noting that SimpleDB has been deprecated, it's not listed in the AWS web console, you can't find docs for it anymore, but if you have a running instance, your service API calls still work. And I think there have been many deprecation warnings since, plus migration messages. But they don't want to break existing clients.
I'm guessing this is a similar case where they want to be really, really sure that it's worth offering the service.
Here are the docs: https://aws.amazon.com/simpledb/
VM Migration is only for maintenance on GCP -- and customers can't control it, just Google.
AWS can hot patch live systems in place without any downtime, so, that's better than a migration (which has a brown out / maintenance period)
Even when you offer bare metal, it’s actually still nice to have nested virt! Otherwise, every node has to be a full sized host. So when you have a K8s cluster or similar with a pile of nodes and want to allow some teams to use it (e.g., Android emulator, firecracker, whatever), it’s really nice not to have to say “okay, this group requires full bare metal hosts that they manage themselves”.
tl;dr: nested virt is still a nice to have so that all your infrastructure looks the same.
Edit: Also, you can trigger migration yourself if you want (gcloud compute instances simulate-maintenance-event), but that's mostly to convince yourself that nothing bad will happen.
AWS's non-bare-metal systems can boot in ~10s with enough tuning.
Their bare-metal systems take tens of minutes to boot.
Nested virtualization would allow scaling up and starting new nodes much faster.
I wouldn't assume that VM migration does not exist in AWS. The overall design and implementation of Google's infrastructure somewhat mandated the development live migration support from day one. AWS was designed and built differently, and some types of events that force live migration in GCE do not exist in AWS.
One specific example from Google's VM Live Migration At Scale paper [1] is "Regular maintenance on the power infrastructure in our data centers requires powering down subsets of machines for extended periods of time". The power infrastructure at AWS is designed to be redundant and concurrently maintainable, which removes a significant need for workload mobility within the datacenter.
Personally, I think it was a very good idea to turn the thing that had to be built to launch into a marketed differentiated feature. But that doesn't mean that AWS doesn't have an ability to live migrate some workloads if it is able to do so without disrupting customers, or if it delivers a better experience than alternatives (e.g., instance degrade notices).