Microsoft/Danger lose Sidekick user data in server failure
forums.t-mobile.com
forums.t-mobile.com
I was one of the first Sidekick customers, long before the MS buyout. It was an amazing email/texting device, far easier and more comfortable to use than a Blackberry. Plus the web browsing was decent, and the UI was simple and elegant. All in all, a really cool phone.
Unfortunately, the first generation of hardware was super flaky. I had five of them die on me in a year, often failing within weeks. Since there wasn't any PC sync tool, the only thing that saved my bacon was the server-side data model. When a phone died, I'd get a replacement, drop in the SIM card and all my stuff would be there same as always.
Back in the day, there was no data sync with your PC. I think they eventually released Outlook syncronization, but most consumers don't use Exchange/Outlook. I would guarantee that a huge number of Sidekick owners (most? all?) have never had any of their phone data anywhere but the Danger servers.
To make matters worse, the Sidekick becomes kind of a brick when the servers are down. I can see a lot of people resetting their device because it's suddenly become unresponsive, just expecting their data to magically come back from the server.
This is an unmitigated disaster.
If I remember correctly, Mark Cuban was a sidekick fanatic. I wonder what his reaction to this debacle is.
Disclosure - I've never owned a Sidekick (and know little about them, other than they are apart of Microsoft's 3 screens and a cloud strategy, yet they don't run Windows Mobile (??)).
Did they fire the whole system staff they used to have? I just can't believe they just know stopped running backups and let it die. Sounds odd, and prone to being sued by T-Mobile quick. (Maybe Microsoft is going after T-Mobile for android?)
The most relevant bits:
"The final operator who is going to be pissed is T-Mobile, who has been just as loyal of a partner to Danger as Sharp has been. I don't know exactly what Microsoft has been telling them, but they have no doubt realized that they've been cut out of this deal in favor of their largest competitor. What's worse is that apparently Microsoft has been lying to them this whole time about the amount of resources that they've been putting behind Sidekick development and support (in reality, it was cut down to a handful of people in Palo Alto managing some contractors in Romania, Ukraine, etc.).
"The reason for the deceit wasn't purely to cover up the development of Pink but also because Microsoft could get more money from T-Mobile for their support contract if T-Mobile thought that there were still hundreds of engineers working on the Sidekick platform. As we saw from their recent embarrassment with Sidekick data outages, that has clearly not been the case for some time."
Now, seriously, Microsoft is not be likely to commit fraud in order to get more money from T-Mobile. But they are perfectly capable of engaging in any unethical behaviour that would prevent T-Mobile from investing more in Android, and, less specifically, of anything that could prevent Android from gaining foothold.
They should really worry because many phones run Windows Mobile only as a kernel and the users seldom see its underpinnings. It would be rather easy to port those front-ends to other platforms. Actually, I would fire the developers who didn't isolate them from the underlying OS as much as possible.
Currently the rumor with the most weight is as follows: Microsoft was upgrading their SAN (Storage Area Network aka the thing that stores all your data) and had hired Hitachi to come in and do it for them. Typically in an upgrade like this, you are expected to make backups of your SAN before the upgrade happens. Microsoft failed to make these backups for some reason. We’re not sure if it was because of the amount of data that would be required, if they didn’t have time to do it, or if they simply forgot. Regardless of why, Microsoft should know better. So Hitachi worked on upgrading the SAN and something went wrong, resulting in it’s destruction. Currently the plan is to try to get the devices that still have personal data on them to sync back to the servers and at least keep the data that users have on their device saved.
http://www.hiptop3.com/archives/what-caused-the-sidekick-fai...
They don't use "big tapes". They plan a disaster recovery architecture and (in the cases I've been involved with) use block streaming protocols to mirror data. Granted, I didn't do the backups, but I'm pretty sure they didn't pay us to bust up those streaming protocols so they could not use them.
In 2002 I lost the data to a small (~200 users) hobby site I built and to this date it still haunts me as my biggest failure. So now I'm taking precautions such as offsite backups, forbidding DELETE for all SQL users, etc., to limit the chance of such a thing happening.
HINT: http://www.newegg.com/Product/ProductList.aspx?Submit=ENE...
Additionally, I suspect we're going to see these customers really hesitate on future cloud-based services.
I am admittedly not a good sysadmin, but I keep an eye on my own "pirate" server (that is, outside the realm of the corporate network) and I have already ordered additional storage because Munin shows that, at present consumption rates, we will run out of storage in about 4 months, crossing the 70% mark in three. I know they will take at about two months for corpotate IT to spend the US$ 100 it costs for a pair of disk-drives and that gives me a month to set-up a soft RAID before I cross the 70% mark.
It's completely unacceptable not to have the storage mirrored and load-balanced on a second SAN. And mind you many SAN vendors even bundle software that makes it easy to do. Even if you don't have the budget to buy another SAN the same size, you can mirror the data rather easily with a bunch of servers, each with a bunch of SATA cards and a bunch of low-budget drives. Redundancy is the key here - if you have a drive that's 10x cheaper than the same storage in the SAN, you can replicate in a RAID1 with 10 drives. If each drive is 30% as reliable as the high-grade disks that go into the SAN, you are well on the winning side.
And I seriously doubt there is that much data. How much? A couple dozen terabytes? That's fits easily under my desk. And quite probably in my discretionary budget.
This is the most shocking part of the article. Is the system really set up so that a failure on the server side causes data on the devices to be deleted? That strikes me as bad design. Yes, any sync system comes with a built-in danger that one endpoint will unexpectedly delete data on another, but if servers goes down the phone should hold onto its data until told otherwise.
In fact, I can almost smell a migration directly to Windows 7 Server or whatever the name they decide to use for that.
Edit: Long live local copies!
Before, if a phone didn't work, it was one of a few things, usually the device or the carrier. Now, it's one of a dozen dozen or more products and services. People couldn't export before. They might be able to now.
A friend of mine once noted the subtle difference between how technology can protect you from failure, but a backup can help protect you from yourself.
It feels like we're moving towards a singularity. Expectations, responsibility and technology.
Giving users the ability to make backups of their data certainly does not mean they will actually do so. How many Gmail users actually keep full backups of their email (using POP or IMAP or whatever)? And if Gmail had a case of massive data loss tomorrow, do you really think saying "well, you could've made a backup just fine" would have somehow made everything okay?
I don't deny that this sort of data loss is pretty awful, but "data liberation" is not the solution.
"Data Liberation", as in educated users, is the solution.
You back up your data because no cares as much about your data as you do (repeat...). It's like basic hygene. In the future, few people will be actual computer experts but everyone will know a few things. Backing up data will be one of them. A large enough density of people who understand this will mean those who will lack excuses - see main story!
I suggest it's unreasonable.
As a cell phone user to my provider, my relationship and contract starts and ends there. Then, it gets complex. And yet, I expect it to work without fail. It seems like it infrastructures are credit default swap complex to the average consumer. This is a bad thing. I don't expect an people to should understand the OSI model. I fully expect we'll find a balance between people trusting others with their information (features/convenience) and having it mirrored/synched without disaster locally (nice to be home).
Inconvenience. Wow. Catastrophe seems to fit better.
There are many ways this could have happened - not that they are excused because of that, but it's not fair to say "not to have backups" until we know the whole story.
In practice, you don't know if you can recover your data unless you migrate the whole company to the backup copy and ask them to check every functionality they use -- every time you make a copy (and rely on them doing 100% coverage). The best you can have during the normal operation is a high probability that what you get is a proper backup, because that would be the case the last time you did a full test (if you ever did). Datasets change, ways of interfacing with data change, etc. Testing is not perfect either, because your tests can be broken or can be reporting false positives.
So yeah... noone can be sure they can recover the data. Does that mean that noone really has a backup?
> Does that mean that noone really has a backup?
No, it means in some abstract sense no one can be 100% sure they have a proper backup. But there are some fairly obvious ways in which you can get that number pretty close to 100%.
-- Linus Torvalds
I'm sorry but in this day and age anyone trusting their data to a device that stores it exclusively on the server deserves what they get.
Some will say "the type of person who uses these devices isn't the type of person to understand where their data is located" but I say that's BS. We're in a data centric world now and I don't think it's asking that much for people to be aware of where their data is kept and to make sure they have some kind of "backup" guarantee.
To Anyone Downvoting This: Let me ask you one question. If someone without a seatbelt gets hit by a drunk driver does the fact that the accident's the drunk driver's fault mean it was ok for them not to wear a seatbelt? Or can someone be responsible for their own irresponsibility even though the damage was caused by a party that was more to blame?
Heck, you trust your money to a bank! You trust your bus driver not to crash the bus, your doctor to properly heal you, and your lawyer to put you out of trouble, you also trust your cook to make the meal you like.
Everyday, you give trust to many individuals and entities to whom you delegate tasks that you may or may not know how to do.
Sidekick screwed up, but I would never blame the ones who trusted them what they(the clients) did is completely normal!
The bank legally owes me my money. The bus driver is guilty of a crime if he crashes the bus. The Doctor is guilty of malpractice. If gmail deletes your account you have absolutely not recourse what so ever. None, Zip. Zilch. Zero.
Sidekick did screw up but it doesn't make the people who lost their data innocent either. You have to take responsibility for that which you value.
Every day there are posts here about how great the Cloud is, and how we should just dump our data there and not have worry about data centers ourselves. Isn't that exactly what was done here - individual handheld users put their personal data into "the Cloud" (in this case, Microsoft's cloud) and then got hosed when the Cloud busted, as happens in traditional Cloud-computing applications. Should we now say that everyone who loses data when EC2 crashes deserves what they get as well?
As for your second point about where users knowing where their data is located, I call BS on that. The majority of people out there over the age of 40 have no concept of the abstract notion of "data", or that it has a location. For them, data is located where they physically access it. Technology is (should be?) about making life simpler and abstracting away bits that people don't need to know about or have the knowledge to make decisions about - for 95% of people, this includes where the contacts on their mobile phone are stored.
If you think people should know where their individual bits of data are, try explaining to my mom how her contacts could live on her SIM card, in her device's memory, and on her desktop computer, and then ask her what the best place is for them.
Same here. Sidekick screwed up. No question. But that doesn't make it ok for the users not to have this data backed up.
2. How about you read my post before commenting on it. I didn't say they did know where their data is I said they had a responsibility to and if they weren't they were being irresponsible.
In the worst case, maybe you'll lose up to 24 hours of changes in case a backup needs to be restored.
For the data to be lost entirely, with no backup whatsoever, simply beggars belief.
If you hand priceless data over to someone who openly tells you they won't back it up they might be the idiot but you're still irresponsible.
Is a lack of a seatbelt the cause of the accident? Would having a seatbelt on prevent the accident? Is the drunk driver not the 'most guilty' party here?
No offense, but this is the type of situation where the company (Microsoft/Danger) needs to really get smacked back into the Stone Age. It's a case of huge neglect if they just lost all of their customers' data. I mean what will probably happen is just an apology with a, 'sorry this won't happen again folks' is wholly inadequate for what they caused.
Reaching into your other comment, while the bus driver and doctor have laws stating that they have legal responsibilities, there seriously needs to be one here. Regardless of consumers' ability to backup their own data, this sort of incompetence is unacceptable from any company operating on this sort of scale.
It really enrages me that the likely outcome of this whole thing will probably be less than a slap on the wrist, if that. While I'm not really a proponent of 'cloud services,' the fact that companies feel that they can offer these services and try to weasel out of any sort of damages they might cause through their incompetence by trying to add things like, "we are no liable for any damages" in some contract pisses me off. This is the same sort of legal nonsense as the 'arbitration' clauses in EULAs where the company is the one that gets to choose the person that will decide who is right and wrong in disputes... and someone somewhere in the company actually believes that this is fair.