Why no company that values their data should ever "Go Google"
e1ven.com
e1ven.com
One way to address these kind of problems is to use cloudHQ to synchronize your Google Docs with SugarSync (clickable: http://cloudHQ.net/sugarsync). (We are releasing dropbox this weekend).
Here is our idea: there are many different cloud storage services and each of them is very good for a particular use case but not so good for other use cases. For example, Google Docs service is excellent for collaboration but not so good for storage. On the other hand, Dropbox or Sugarsync service is excellent for storage but not so for collaboration.
So if data is continuously synchronized between Google Docs and lets say SugarSync storage, then you can always fall back to SugarSync when Google Docs becomes unavailable and vice versa. In order words, you can use other service as a standby site in case of service unavailability, malicious or accidental deletion of files, etc.
He talked me through all the design considerations they had to do for their backup system for the datacenter here. It's a lot of work, but there is a lot of business value in having a working backup system.
But Google Apps is a pain in the butt to backup, since we don't have direct file access to things. The best strategy I can see, if you want to stick with it, is to email yourself every file, every edit. That's pretty insanely tedious.
Automatic daily Google Apps backups -- all of it, even Google Sites -- for just $3 per user per month. Unlimited storage. One-click restore for Google Docs and Gmail messages. And we're soon rolling out full-domain search, so you can query all your Gmail, Docs and Calendar accounts simultaneously.
We also backup your Twitter Feed, your Facebook Pages and your Flickr or Picasa photo albums, should the need arise.
Also, for the record, we have a 1-800 number that we actually answer, so in the event you have trouble with our service, we'll pick up the phone and talk you through the fix. We can't repair your Apps account, but we can darn well make sure you get to the data that's in it. We wouldn't charge you, otherwise.
Do you have any integration (or plans for integration) with Dropbox? I'd love to sign up to a service that simply dumps a copy of all my backed up files to a Dropbox folder somewhere, once a week.
Our Pro 500 plan is $19.99 per month and includes 10 Google Apps users and unlimited storage. That actually breaks down to less than $2 per user per month. Beyond that first ten, additional users are $3 per user per month. I usually just quote the $3 figure for simplicity's sake, especially since most admins using a paid Google Apps domain (Google Apps Premium) have more than 10 users anyway.
[1] http://code.google.com/googleapps/appsscript/service_docslis...
But he's an operations guy like me, and he probably wears both a belt and suspenders just in case the belt breaks. So in response to the complaint that emailing yourself files to insure you have copies is a pain, I suggested he script the moral equivalent using Google's provided APIs to the docs service.
I know you can use pop (and maybe imap) to pull email out of GMail. but can you recommend a product that can handle slurping and backing up all the mail for your domain?
You can also do it direct from the web UI. Just select all your documents from the Google Docs homescreen, click on the "Download..." link in the righthand column, and it will ask you what formats you want then generate a zip file to download.
I don't know of a command line script that uses OAuth (GDocsBackup has a command line mode but uses password auth.) Anyone?
There are plenty of apps on the marketplace. I assume they're all automated.
There is probably a business idea here.
Shameless plug: This is why I wrote my app, CloudPull. CloudPull backs up your Google Docs, Calendars, and Contacts to your Mac. If for any reason you are unable to access your Google account, CloudPull ensures that you have backups of your Google data on your Mac.
More information about CloudPull is available at: http://www.goldenhillsoftware.com/
The real story here is that a paying customer was unable to get anything better than a free user.
FWIW, I must be lazy or something, but I don't think the user of a VERSIONED online document storage service should be reasonably be expected to also have his/her own backups. What the hell am I paying for then?
Why can't there be two stories?
If I park my car and don't lock the doors, and come back to find it gone, is the story that I'm an idiot for leaving a car unlocked, or is the story that the neighborhood is unsafe?
They're both problems, but I can actually solve one of them myself. That doesn't excuse the $BAD_THING that happened, but it does offer me options for when it inevitably does.
So in your analogy: You park your car at a garage with a guard, pay them to keep it safe and leave your car unlocked. And when you come back it's gone.
Yes he could have done something to make it less likely it, but he also explicitly paid someone else to handle it for him.
Because it seems to be a foreign concept in this day and age, I will repeat it again.
You always backup.
I entirely agree- When things are mission critical to you, you need them in an environment where you can control it, back it up properly, and ensure it stays up.
We're not a priority to Google (Understandably.)
For things that really matter to your company, you should bring them in-house, so you can ENSURE they're backed up, and available when necessary. At least that way when things do fail, you can make fixing it a top priority, rather than a nice-to-have.
The reality is that Google is much less likely to damage or lose your data than you are, but it's always good to keep extra copies around locally. This is important for many reasons, like access when either Google or your whole net connection is unavailable, and more flexibility with the data.
In-house IT fails plenty of the time also, and plenty of organization cannot afford to fix some failures. One must consider the probabilities in all cases.
Personally, I think this applies to yourself, too. As much as possible, make it difficult for you to overwrite your own backups before they expire. If possible, encrypt and outsource.
Google doesn't do human, it doesn't matter if it's a free gmail account, a serious-money adwords account or your entire business on apps, you're just a number if something goes wrong you're on your own, shouting at the unmoving monolith.
It probably would have been better to call, I mean you are paying for 24/7 phone and email support.
I haven't had any problems with getting support through Google Apps for Business, I might be an exception though.
The problem is that your mail is crucially important to you, but rather less important to Google. In such a scenario, it makes sense to change your plan to something under your control, even if it's painful.
There are providers that can give you various guarantees, response times and that will make key personnel available to you (at a price). Google is not one of those providers.
We got back several emails explaining that they are very sorry,
but while they are working on retrieving the file, they can’t
promise any dates or times.
If you really, actually do value the data, then the best way to assure its safety is to take control of it. One company I worked had a practice of taking a set of offsite backups to an emergency facility, restoring the data and doing some sample processing. Every month they did this, without fail. That's caring about your data.
These are not binary decisions. People vastly overestimate the value of doing things in-house. Most of the candidates for those "in-house" data management jobs couldn't cut it at Google. Why not leverage Google's HR and recruitment, training, and expertise in infrastructure operations to your benefit, esp. when you consider:
(1) that your own hires offer you no better protection; and
(2) backup solutions are necessarily imperfect.
There's a lot that you can do about your employee that you can't to with Google. You can educate them, align their motivations with yours, etc. You can review the process and make sure it doesn't depend on one person. The list goes on.
Of course there are times to outsource. Times when practicality makes it necessary, when money makes it desirable, and so on. I do not take issue with the very idea. Sometimes it's the right choice. But sometimes it's the wrong choice.
I'd rather have a 1% chance of losing my valuable data if I feel in control than a 0.1% chance of losing it while feeling like there is nothing I can do about it.
The problem is just having your email stored outside your private network.
It makes a lot of sense for startups and small companies, less so when they start growing and they can afford a staff of sysadmins.
There's nothing magical about your "private network". These days corporate networks are chock full of holes and ripe with hypocritical security priorities.
Use Google apps for business ($50/year/user). Hire one of the many companies that archive Google Apps data.
It is total folly to think that you'll buy better security by paying for your own infrastructure and a small army to run it. That's like claiming the only safe automobile is one that you and your friends build.
The only way to ensure that you never have downtime is to not use the Internet, and keep all your servers connected to a nuclear reactor, all inside a mountain. That covers power and not being flooded or hit by a tornado, anyway.
Even then you'd better be careful not to trip over any of the power cables or type "rm -rf /" as root.
It's now been over 8 hours since it went down, and no fix from Google yet. 4+ days on the missing file.
If I was running in house, I could have entirely restored the mail server from tape by now. I could have swapped over to a hot-spare in a few minutes. I could have failed over to our backup internet service. I have a lot of options.
With Google, my option is to wait.. And hope my business doesn't lose too much money while Google gets around to fixing it.
http://don.blogs.smugmug.com/2007/01/30/amazon-s3-outages-sl...
"""So what are we doing differently? Simple. Amazon serves as “cold storage” where everyone’s valuable photos go to live in safety. Our own storage clusters are now “hot storage” for photos that need to be served up fast and furious to the millions of unique visitors we get every day. That’s a bit of an oversimplification of our architecture, as you can imagine, but it’s mostly accurate."""
You can always maintain a hot-backup, fail-over of your site on your own servers -- perhaps with reduced functionality until the scalable cloud services come back online. For a mission-critical site, this would seem to be a reasonable tradeoff.
1. Use a resilient external DNS system 2. Configure local mail clients to save messages for "offline browsing" or equivalent. 3. Use local mail client to check mail at least once a day
When email goes down, point the DNS to a pre-configured mail server. You have all your archives and shouldn't be missing any mail since delivery failures will try again not too much later.
As for the Google Docs issue, ensure that users are exporting vital docs frequently, preferably when they're done making a change. You should actually edit locally and use git. You can set it up to automatically upload changes to gdocs on push.
As it is, it feels like my requests are going into a blackhole. This is very important to us, but to Google, we're just another tiny startup.
I'm turning more and more to plaintext/markdown on Dropbox.
In short: if you're reliant on the cloud service for your backups (as distinct from your sync'ing), you're at risk.
AeroFS is the best off-site solution so far, but it's still very beta.
Support and reliability
24/7 phone and email support for critical issues
99.9% uptime guarantee SLA
4+ days on the missing file, 8+ hours on broken email.
Has anyone tried those products?
It's a big deal to us, not a big deal to them. We'd pay extra for a tighter SLA if that option were available, but as it is, on outage hurts us a LOT, and them not very much.
That would seem to be prescription for bringing it in-house, despite how little I want to do that.
I would not consider running my own services and not back up my data. If you use GMail and/or Docs for free or as a paid customer, still, why would you not make local backups?
A lot of businesses are completely dependent on Google via organic search and AdWords for 90%+ percent of their revenues (without even realizing it).
What happens when you log into your AdWords account and you’re greeted by this cryptic message:
http://www.devside.net/images/adwords-account-suspended.png
Except that you've done nothing wrong, have never been warned, are not a spammer, and are completely legitimate.
What happens is that after contacting Google you soon realize that a monopoly with no customer support is the most dangerous one.
Come back and tell me that relying on using Google docs while keeping local backups is problematic or awkward. That would be a fair complaint.
Toshiba isn't going to help me if my drive goes down either. Whether they send a nice or not nice email isn't the point.
Ultimately you are going to rely on someone else and something else. Be it your IT guy, the company that hosts your backups, the power company, the company that makes the machines you keep your backups on, etc. You can distribute this so that you rely on lots of someone else's and any one or more of them can fail, but often you'll still have one point of failure somewhere (the guy who monitors the distributed backups for example).
I mean each organization needs to have in-organization backups and each individual needs to back-up their machine.
Yes, my IT guy. I need to have an institutional relation with my backup guy, not just a paying relation. The IT guy could send the data to three different online backup services but me having a personal and institutional relation with him makes sure that whatever scheme he uses will remain reliable as other things change.
The point is that every organization has at least one mission, one task that it is worth doing within the organization rather than offloading to another entity. And backup is always going to be close to that mission.
The UI in a lot of places makes no sense. For instance, there are settings for when you set up an email list that are labeled so obtusely that you're not really sure of what you're selecting.
Another thing that totally bit us once was when dealing with document ownership in Google Docs -- a lady in marketing had created a bunch of documents, then later left the company. After scouring the back end for about an hour (literally), I concluded that there was NO WAY for the admin user to reassign the owner of the document to a different person. I ended up having to log in as that person (luckily we kept people's accounts active for 6-12 months after they left) and switching the owner that way. Totally absurd. (If I'm wrong about that, please do tell.)
As far as customer service goes, it's definitely a minus. They don't make it easy to tell you who to call or email when you do have a problem, and once you get there, the time before your problem is resolved can vary wildly (from hours to days). That really sucks when you have a salesman breathing down your neck because he can't use his email.
For Email, we plan to, but it's a bit more complicated.
I'll need to imapsync each account, so it's going to be messy, so I'll need a week or two to plan it out, and build/test a new Zimbra server.
Always have redundancy too. We keep most of our files in Dropbox. But we all use Time Machine too. We also use version control for a lot of our work.
This might seem like overkill, but each has a specific use and purpose with the added bonus of providing us with backup redundancy.
The only real solution is backup, and the only backup that can truly be relied upon is one you make yourself; in which case the point in making use of a cloud service in the first place is pretty much reduced.
if you open Excel and put some typical numbers into it (like commodity hardware cost, networks, electricity, data center mortgage, basic maintenance employees, 40% margin, etc...) you'd see that $50/user/month is just basic usage - timeslice to run provided software on provided hardware. There is no room in that number for a real person dealing with your specific issue. An elephant can't squeeze into a needle eye, even if he promised it to you in writing and accepted the money - just look at the elephant and at the needle with your own eyes.