Company was hosting cardiac patient monitoring on EC2
forums.aws.amazon.com
forums.aws.amazon.com
Remote cardiac monitoring isn't for people who are imminently going to die of catastrophic heart attacks or who risk dropping into a fatal arrhythmia (wonky heart rhythm). In fact, there isn't even solid evidence that it saves lives. What it is used for is investigating patients who have vague symptoms which might be related to transient changes in their heart beat.
These are people who come to their doctor at the age of 54 and say "I felt my heart beating really quickly and felt kind of faint" or "I felt kind of dizzy and then passed out. It's happened twice in the last month." The more traditional way to investigate these patients are with what's called a Holter monitor, as alluded to by a previous poster. These are little belt packs you carry around while wired-up and that record your ECG for 24-48 hours at a time. The main weakness? You said it: the device only has a 24-48 hour windows to capture the weird, often rare, rhythm.
There are different ways remote cardiac monitoring systems report their results and I'm not sure which this particular company was using, but it doesn't really matter. Some of them only report weird stretches, others only report events when the patient says they're feeling symptom X, others are reporting continuously.
Take-home message: this is not life-or-death data. When doctors (at least those who are allowed to keep practicing) think a patient needs critical cardiac monitoring, they admit them to hospital.
This was a tech guy, looking to jump the queue by trying to raise a red flag because his servers were being used for--OMFG!--cardiac monitoring. A lot of my doctor buddies use similar strategies when they're caught speeding by the police. "I just got called in to the hospital!"
…to see a patient with a really nasty rash.
b) Good to know from responses that amazon's paid support is crap. They'll officially tell you "were on it".
c) People are bashing this person for relying on amazon not failing all at once. I would be surprised if they hosted on amazon relying on each individual node being up 99.98% of the time. That is crazy. You build on cloud computing knowing that each one node can fail at any moment, but you build it to fall over to another node. From experience, sometimes nodes just... die... or get really unstable.
d) People bashing the dude vs offering future advice. I'm sure if his company is in deep shit, he's well aware of it.
Finally, I'd ask why if they were running a business that could generate / lose thousands of dollars in a few minutes why they had chosen a residential internet service instead of opting for the business package and why they did not have a backup connection as neither the business packages or residential packages came with a SLA that was suitable for the type of risk inherent in their business.
If their data is so important why are they hosting it on the absolute cheapest and shittiest servers they can find without a backup solution in place? If they read the TOS for EC2 it probably says not to run critical infrastructure on it.
Perhaps for sensitive data upon which people's lives depend they might want to look into something like:http://h20223.www2.hp.com/nonstopcomputing/cache/76385-0-0-0...
Yes, it costs more than EC2, but there's a reason for that. Wait until this company finds out that if their values exceed what MySQL expects it will silently truncate it.
If this is a serious post and there really is a service that monitors cardiac patients from home, first you would have to deal with the unreliability of in-home broadband connections. What if a router needs to be reset, or the DSL goes out? Even if you ignore that, you would expect that any server setup will go down at some time. It has happened to Google, Facebook, Twitter, and now AWS and PlayStation Network. I remember when Rackspace went down a few years back (a truck crashed into the transformer that powered the datacenter - http://www.datacenterknowledge.com/archives/2007/11/13/truck...). All of Armenia was knocked offline when a woman cut a fiber optic line while digging for scrap metal. I'm not sure I'd trust the internet with any kind of life or death monitoring.
I think some IT person was freaking out that they had downtime and slightly exaggerated the life endangering part, they likely just lost information that may have been used in a future diagnostic manner.
"The Remote Health Monitoring System is the IT backbone that supports HealthFrontier’s entire portfolio of solutions, including the ecg@home™ and the microtel™/ecgAnywhere™. It captures data transmitted by the patient through USB, Bluetooth, or trans-telephonically. The RHMS™ then stores the information in a database, which can be accessed by the patient’s doctor through a web-based interface. Benefits include:
Cloud computing: The RHMS is hosted on a network of servers across the US, which increases reliability and eliminates the need for a large capital investment in on-site hardware and maintenance."
http://www.healthfrontier.com/rhms.php
How many other companies provide home ECG monitoring which report over the web? I'm looking but having a hard time finding more. I will update this post as I do.
[UPDATE] Other Home-based ECG monitoring services:
Medtronic CareLink: http://www.medtronic.com/your-health/heart-failure/device/ca...
[UPDATE] A more complete listing: http://www.medcompare.com/matrix/1475/Outpatient-Cardiac-Mon...
The company I was most impressed with was http://www.halomonitoring.com . The device seems to be designed well and the monitoring services (online portal for caregivers or family members) are impressive, comparatively.
There's kind of a sea change happening with home health monitoring -- from the "I've fallen and can't get up" devices to more robust and interactive solutions. GE recently bought a company that provides institutional monitoring equipment and services.
Lemme know if you want the full report :)
In addition these devices can be used to take 12-lead EKG measurements in transit (e.g. inside the ambulance) and send them to the doctor for analysis before the patient arrives.
I'm very surprised that they're hosting these services on EC2. The company I worked for recently finished building a huge highly-redundant data center specifically designed for hosting this kind of medically-sensitive information. I'm willing to bet it was a bit more fortified than even Amazon's.
EC2 just shouldn't be ALL of the servers.
Servers are not physical things that you can just go get when there are problems. They are imaginary devices that can suddenly vaporize, leaving no trace. Plan your deployment accordingly, and you won't need to be writing emails saying that people are dying because your data center blew up. Your systems will silently switch over to some other data center and you'll file a claim with your insurance company to have the servers replaced on Monday morning.
In other words, "the cloud" had nothing to do with this problem. It could happen if the server was physically attached to you or in a abandoned missle silo with an army of armed guards.
Yes, for something life threatening like this, EC2 is a bad idea, but they didn't even bother to take advantage of the geographic redundancy, much less something so basic as having backup AMIs ready in another AZ in your current region.
Of course, a whole lot of companies are learning this now.
In fact, it's looking like all the automatic-failovers from people hosting on those data centers compounded the problem when everyone's scripts tried to recover at the same time – what was referred to in a previous article as a "bank run".
But yeah, if you're doing heart monitoring, you need to have your servers replicated across a wide geographic area.
As for backing up to multiple regions, I can imagine them thinking that sending everything over the public internet as being a bad idea. However, not having a multi-region/multi-provider failover plan was a worse one.
I wonder if this will get more companies to maintain a non-cloud, vanilla setup on some dedicated or collocated boxes.
May be we should have an annual day to bring down our primary servers and see how the backups do. The idea shouldn't be to confirm if everything works or not. It should be to determine what did work and did not work.
Maybe you could even timeshare your machines out for some other instances for a discount on your service, similar to how you can send solar power back up the power grid for an electricity credit. Then ec2 really WOULD be a cloud!
Amazon could even simply benchmark your server and network connection, and compensate you as a function of that, which would drastically reduce the incentive to hook up old crappy hardware to the cloud.
It's really the idea of being able to run copies of your OWN instances locally that intrigues me though.
No reason you can't already do this.
Just rather than supplying the hardware you have to rent it from Amazon.
Would be interesting to know how these instances were affected by the outage. Since the problem was with the EBS infrastructure not the actual compute hosts I'm guessing this wouldn't have helped any
"Eucalyptus is a software platform for the implementation of private cloud computing on computer clusters. There is an enterprise edition and an open-source edition. Currently, it exports a user-facing interface that is compatible with the Amazon EC2 and S3 services but the platform is modularized so that it can support a set of different interfaces simultaneously. "
One of the features that allows AWS to scale is that it runs on a unified and consistant hardware platform. Take a look at the recent Google Data Center video or the Facebook Open Server initiative - companies try where possible to run identical hardware for all of their needs as it removes inconsistances in behavior and performance between arbitrarily different pieces of equipment.
So the idea of giving Amazon your own hardware to host is terrible.
The other part of his idea sounds like mirroring servers so that if an EC2 instance goes down, some bare-metal hardware running elsewhere could kick in. That's totally feasible, but no need to do it in the same DC under Amazon's control. Just set up your own machines (bare-metal or virtual) and keep them in sync with your EC2 instance. In the event of EC2 going down you can just change your DNS over to point your fall-back IP or do some clever off-site IP load-balancing tricks (assuming the load balancer hasn't gone down too)
In fact this would have been a good strategy for the ECG monitoring folks if they needed to keep service up even during a prolonged EC2 outage.
Without any background, it is really tough to tell where this fits. You can't blame the people for making the wrong decision until you know what the risk profile for the failure is.
"This not just some social network website issue, but a serious threat to peoples lives!"
Sometime during the last couple of days, a lawyer was getting a lecture from IT on how the system works and is going to have a very, very bad year.
Still, not really forgivable to be 100% AWS, but it's a very different story if this is primarily a recording device vs. a monitoring device.
Seriously? I know there's HIPAA standards for securing sensitive health records, but aren't there standards for system up-times?
Given the information we had a week ago, I can see how Amazon may have looked like the best solution for HA. Their site never goes down, after all.
Second of all, according to Amazon PR speak this failure should not be possible. Every person in IT worth their salt should know PR speak is not to be trusted. Being reliant on AWS for your average business app is completely okay though. However, hosting apps which might endanger peoples lives on black box technology over which you have little influence is moronic imo.
Last but not least I seriously question the privacy implications of this. Hosting parts of your medical app in the cloud is fine and I don't know the details of this app, but I wouldn't be happy to know that my medical history is stored by some third party I don't entirely trust with such sensitive data.
2) Amazon is HIPAA certified, which means they adhere to the industry standard for storing private medical data. In this sense, they are as good as any other company that such data is being passed around.
Didn't know what HIPAA meant, but apparently they are certified for storing medical stuff.
Up to a few months ago we had a ASP.NET shared hosting customer that was doing some kind of data relay web service for medical imaging. No encryption. Patient data in full view on the server. No redundancy. Apparently it was used for outsourcing imagery review or something. If it didn't work doctors would have to drive in from home which slowed down the diagnostic process.
"Mission critical" on a $30 a month shared hosting plan. Very much not HIPPA compliant.
In my experience, small companies typically have one hosting provider, off-site backups, and the probable but untested ability to bring up a fresh installation in a few hours. That's both common in the real world and good enough for most situations. It would get you through an outage like this.
The poster on Amazon's forum misrepresented the severity of the infrastructure and tries to backpedal later on in the thread. This is an example of someone using EC2 and fucking up, firstly for hosting production level infrastructure without backups, and secondly for misrepresenting the importance of the infrastructure in an attempt to get more technical support, making them look liable for some sort of medical malpractice.
The poster probably panicked when they realised their first mistake, and thought they were showing that they were trying to get back in control. I don't think the poster was totally stupid, maybe she/he didn't understand the risks of EC2, or maybe it wasn't even their decision to have no backups --- in which case they should have pressed harder at meetings or refused to continue to work on the project. And then they panicked. We can all learn something from this.
If they are monitoring cardiac patients, they should, at least for non-critical people, have the instrumentation cache the data and send when available over leased lines.
If it is critical, it should be PROPER HARDWARE attached to the POTS network (like we have in the UK!) with multiple redundant networks over several distinct carriers and multiple monitoring stations.
It sounds like their product is a) crap and b) dangerous if used for critical care.
Now, if zero downtime is the requirement, then you just need to have a lot of data centers and well-tested failover procedures. It's all very simple, and EC2 can easily be one of those redundant data centers.
But even for non-critical medical data, non-tech savvy medical folks (and this includes some physicians, although in general physicians are more tech-savvy than most managed care decision makers) are going to get into increasingly serious trouble if they don't think through these kinds of things very carefully.
Driven by electronic health care mandates and incentives, companies are pushing portable devices for medical data gathering (portable as in tablet, ie without local data store), and some groups are going to be knee-jerk tempted into buying into this without thinking through the ramifications of not owning your own data -- and by that, I don't mean vague marketing-speak hand-waving that "you own your data" through the simple mechanism of password-protected access to your patient's data living on someone's server farm.
By actually owning your patient's data, I mean strongly considering restricting your local practice to use of on-site media.
If you don't, you'd better have complete confidence that putting your patient's data on the web reflects a reasonable standard of care/privacy, that retention policies don't bite you in six months when your cloud provider goes out of business (or changes hands or farms out data storage or backup to some "economical" third-party), that you're going to have access to your data when you need it, that your cloud provider is going to audit access and changes to your patient's data and report the results of that audit to you accurately and regularly, that legal requests for data are not acted upon without you or your patient's consent, that backups are subjected to the same levels of security, etc. etc.
Electronic medical records are a good idea. It can be done right, as systems such as the Veteran's Administration's VISTA system have demonstrated. However, there is a fine line to walk here. Implementing EHR for the sake of government incentive, or for the convenience of insurance companies, or even the convenience of walking into an exam room carrying a light-weight tablet (which have their own disadvantages for data gathering) rather than a laptop with a hard-drive isn't going to cut it when you put your patients and your practice at risk of harm.
If it can go wrong it will go wrong.
Of course, then things will swing wildly in the other direction with massive legislation and regulation thats designed to get a lawmaker reelected more than it is to save lives.