How upgrading to Google Apps for Business killed my company's email for 6 hours
floriancornu.blogspot.com
floriancornu.blogspot.com
If you aren't fully addicted to the Gmail interface, there are better options for business mailbox hosting.
Best of both worlds.
Think about failure like resistance. Points of failure in parallel gives less overall risk of failure. Points of failure in serial (like adding a front-end) gives greater overall risk of failure.
- Having webmail which is not protected with two factor authentication
- Enabling POP/IMAP which also does not have two factor authentication
What do you mean?
Forward to your firstname.lastname@gmail
Configure SMTP
Configure filters so you have a folder for company email
problem solved
It's a difficult compromise between availability and security/privacy. I want to be able to get at my mail from all my devices via any Internet connection, but. I still want privacy from dragnet intercept-and-record-everything surveillance.
My current plan - a mail server in the least US influenced jurisdiction I can manage, automatically encrypting all (unencrypted) email to a public key of mine, the forward the encrypted email to a gmail account. I can decrypt mail at the client on my phone/tablet/laptop/desktop, but I lose a lot of mail searchability this way.
Using Google Apps, all factors considered, seems to be less reliable than the old free Gmail. (Since Google can't pull the plug on you like it does to its paying customers, like this case)
Google can easily pull the plug on any Gmail account they want. It says so right in the terms of service. You have no guarantee of service or access to your data. Even higher profile reporters have had access to all their Google data (gmail, calendar, contacts, documents, google drive, etc) disabled for having what appeared to be client payment data in a single Google doc. They were only able to restore it after a few days because they had some connections.
My only gripe is the lack of support for two-factor authentication, apparently this is under development currently.
Otherwise I love them.
What I'd like is a service that kept my emails encrypted on-disk with my user password. Not because I think emails are private -- they're not in the sense that I fully expect someone could intercept them in transit, even if that would be illegal -- but just because it would give reassurances that my emails can't be data-mined for advertising purposes à la gmail, or if the mailserver had a major security vulnerability, a hacker couldn't copy all my emails wholesale.
But I don't know of any service that provides this, nor of a way to do it myself easily (any mailserver I set up myself would probably have far more vulnerabilities than one I rent from someone else!).
Thanks for posting this!
If my email goes down, getting a partial or full refund on my $2/month mailbox fee is the least of my concerns.
Then -- and this is where they destroyed my trust in their service -- I took a look at the status panel the following day, and they cleared the history of the downtime (it shows 30 days of issues). Instead of showing the issue, they marked it as green and deleted the messages describing the issue.
That day, I stopped using Rackspace for everything. (This was about a month ago)
Still disappointed they don't send out emails when there's downtime..
Would probably use them again the future. (This was my one negative experience with them, btw)
They really screwed the pooch, here.
As someone who has been in IT for 20 years, I can tell you're in IT. This type of semantics drives business users crazy.
Business User: Hey my system is down!
IT Person: No, it's not down.
Business User: I can't log in to it.
IT Person: We are doing scheduled maintenance so the system is unavailable, but it's planned maintenance so it's not down.
Outage suggests temporary and regrettable.
This is more like Google just straight up fucking people over via neglect. I was suggesting that stronger negative terms be used.
For us, accounts that were forwarding mail someplace else continued to work, but anyone that had to log into Thinkful's gmail account didn't.
The time delay is ridiculous and unacceptable, but it sounds like customer support did the most they can possibly do in that case: Engineering had implemented a (questionable) migration policy, and there are no toggles or switches that customer service can fiddle with to make it happen more quickly.
Many/most parties could probably cope with it reasonably well -- if they were told in advance that it would occur.
Upgrade over the weekend. Let customers know your systems will be experiencing some planned downtime. Etc.
Come on, Google. You claim to be supporting businesses. Well, act like it! Among other things, this means communicate, before the sh-t hits the fan.
----
P.S. This situation isn't even asking for a dialog. Just Google's communicating some information -- what's really gonna happen -- up front.
Sometimes, your marketing needs to take a bit of a hit, for the sake of your customer.
I've little doubt that most of the engineers there respect this. We have to deal with such situations every day.
If I were particularly direct, I might suggest one of them go over to marketing with his/her "stomping" boots on, until they come to understand this point.
That this was happening to Google, and that it occurred exactly when we tried to _pay them money_ definitely makes the situation hilarious.
Google's support is a complete joke. The only way to get things fixed in a timely manner (if at all) is to ask someone who works for Google to escalate internally. Then you stand a chance. Otherwise you're SOL.
On top of that the Gmail team routinely ignores internal feedback before rolling out new features or UI changes claiming the Googlers do not represent average users.
I would recommend the OP to contact a Googler friend if they have one.
Google's arrogance and disrespect for its users will be its undoing.
Sorry, but that isn't even close to a solution. The problem here is that a business-critical facility was off-line for pretty much an entire business day, at who-knows-what cost to the business.
Many small businesses that use facilities like this are trading internationally or otherwise need 24-hour functionality, so the idea that you could just schedule something to "minimize the inconvenience" doesn't really help them at all.
I have other clients who wouldn't dream of paying $5 per user per month, because they get free email forwarding with their domain name or cheapo web hosting.
I have _many many_ clients, who if I gave then sufficient notice and said "we're upgrading some infrastructure, we need to take the mail server down for up to 24 hrs, does after close of business the Friday after next suit you, or would you prefer we did it Sat night/Sun morning?", would be _perfectly_ happy. (with the possible exception along the lines of "oh, we're at a trade show that week, can you move it forward or wait till a week later?")
I wonder how big the crossover is between companies who really absolutely _have_ to have 24/7/365 email uptime at better tha 3 nines, but who haven't already got something more robust in place than a handful of personal gmail accounts?
I can't help but think they're "doin it wrong"…
The problem here appears to be that Google's pathway to get to such functionality -- assuming a service branded "Google Apps for Business" does in fact offer a higher standard of service than three 9s and isn't pure marketing spin -- is broken in a way that could undermine an organisation that wanted to make that change.
Obviously that won't affect all businesses, but if you're in that boat then it seems Google can't even help you get to their better services without risking exactly the kind of interruption that a more serious/professional service is supposed to avoid. That seems bizarre to me, and doesn't speak well of how they run their business-level services. Combined with their recent trend of killing off even moderately popular services altogether, it doesn't exactly fill you with confidence about trusting Google with anything your business really depends on.
The free version was available for a year before the paid version was made available.
As far as I can see it's precisely the upgrade path to get them off that free service and onto a higher level that is broken here.
Surprise, surprise on the day the account totally shut down. I called in a panic and after a lot of fiddling about (and reassuring me that they had recorded that I thought there might be a problem) they got it back up with everyone's mail after almost a whole day of downtime.
To add insult to injury Google don't run a charity program in the UK despite the non-profit laws being a lot stricter than the US and it being easy to validate a charity - so they're paying full rate for Google Apps for Business.
Thanks!
I asked the organisation, and found out that it's not an official programme in the UK. The only one they have here is Google Grants for AdWords - http://www.google.co.uk/grants/.
Basically, an enterprising IT volunteer had set them up with vanilla Google Apps back when it was still free for small business and had a 50 user limit, and they just got grandfathered in to the new Apps for Business. They've never had any problems with the user or space limit since they just use the setup for email, analytics and AdWords campaigns and only have about 4-6 people "working" there.
Apologies again.
Here's a link comparing the free edition with Google Apps for Business: http://support.google.com/a/bin/answer.py?hl=en&answer=17512...
Thanks for getting back to us though!
https://support.google.com/nonprofits/answer/1614602?hl=en&t...
If you could ask someone involved in that charity how they got free Google Apps I'd appreciate it as I work for a number of charities and the costs of Google Apps are a significant admin overhead. I've tried calling the sales people and they say it's coming soon but they've been saying that for over 5 years now.
/end{rant}
Generally speaking the web interface being down, or the imap server being bonkers, was very rarely if ever a problem.
The biggest suggestion I have for anyone wanting to use the paid Google Apps is to go through an authorized reseller. You will get better pricing, and they generally have faster access to higher level support than going directly through Google. I had an absolute nightmare of a time with client support (construction workers with little computer literacy) but any legitimate problems during the transition were quickly dealt with by Google via the reseller.
Also, good luck getting a response from the Google Apps sales team in the first place.
Agree that the transition took ENTIRELY too long.
Author was awake and saw some horrible things.
The post is dated May 27, is Google planning to announce a new feature for Apps this week and this is some sort of a preemptive PR attack?
Given a company larger than 10 might actually upgrade (small amount of shared email accounts) I could see it causing big problems for some.
Sure there is phone calls etc, but emails are used when you need an actual record of something. Also relying on mailservers to not bounce the emails and actually deliver in the end is laughable, when mailservers do go down this hardly ever happens properly.
If you need a record of something, I do not think that unsigned emails are legally binding anyway, again you should probably submit signed contracts over the web not email.
If it is not time critical, email can work, but the OP said a six hour outage would be a problem, and I still think that is one they have created themselves and is a business risk.
If it is that critical do you have 99.99%+ (52 minutes a year) uptime guarantees on your mail service? That is what you are asking for, and to actually deliver that (rather than an empty SLA promise) is something that very few businesses actually try to work to, especially small companies. Gmail certainly does not try to provide this.
For lots of people in different roles, email is an essential tool for getting work done. Not everyone has a role that can be readily translated into an API. Business Dev/Sales for example, depends on email to communicate with the various folks that they need to engage in. Whatever those folks do, it ends with the company getting a check, so it's important.
Generally speaking, it is pretty inexpensive to deliver a 99.9% available mailbox with a 100% guarantee of external mail delivery. The fact that Google bungles a conversion from free to paid service so poorly is a sad statement when they are supposed to be a shiny alternative to the traditional Micrsoft messaging stack.
I have known large (email dependent) businesses have 2 day outages on the traditional Microsoft mail stack too. There are coping strategies. It is annoying not mission critical.
Depending on the business sector of course. If you're a stereotypical weekday business then a 6 hour outage at 9am on a weekday would be a disaster, but a 6 hour outage from midnight to 6am on Sunday morning wouldn't even be blinked at because no one cares.
1. There is no indication in the upgrade process that the upgrade would "take a few hours to complete". Quite the opposite, it indicated there would be no downtime.
2. The customer service reps seems to have little insight into what was happening with the account which is a bit scary. I'm always nervous of black hole "processing windows" where all you can do is wait and hope for the best. It's bad when it is customer facing, it's worse when it is customer service facing.
3. Upgrading on the weekend could potentially be more troubling given that customer service may not be staffed with the quantity/quality to fix issues if they occur. Not saying this is or is not the case for Google, but running into problems on a Saturday morning and getting a "call us back during normal business hours" is just as troublesome.
What should have happened in my opinion?
1. Google should document that there can be a temporary delay of x - x hours while the upgrade happens.
2. Florian should have scheduled the upgrade ahead of time with the team, letting them know that worse case there may be an outage of 'x' minutes/hours/days.
3. Google customer service should have better tools as to upgrade process for both the admin (customer) and customer service rep so that it is not a black hole of 'wait and hope'. Even a step 1 of x, or you are number # in queue, or estimated time, etc.
What law of physics necessitates that such an upgrade takes 6 hours?
the one where the customer isn't paying enough money to have somebody working 24/7?
If your business depends on a particular service, you don't accept something vague - you get things signed in contract with legally enforcible SLA's. If you can't afford that, then you just have to live with shitty service.
Except when they do. Lesson learned.
We recently upgraded to paid Google Apps for Business and we didn't get any downtime. This is how it is supposed to work.
Unfortunately we also had irritating problems after the upgrade. We upgraded because our customer support email address had run out of mail quota. Paid google apps has a 10x higher quota, but upgrading to premium didn't increase it.
After ringing customer service they refused to increase it immediately and said that it would be increased "in a couple of months time".
Meanwhile hundreds of our customers were going unanswered.
I suppose I can imagine a scenario in which they would want to wait until after credit card chargeback window before lifting a quota limit, but their support department should understand the paralysis of a business offer to solve the problem.
We had to create a temporary support2@mycompany.com email and manually port email across which wasn't fun and played hell with our ticketing system.
I can't imagine you have 5 gigs (or however many they give you these days) of tickets that can't deal with a few days of cold storage.
Even if we buy that this is more than an administrative change and it somehow moves to better hardware, this is a problem that I would think that Google would have built to a mostly transparent process -- at most long term archives are unavailable after a very brief initial migration, etc.
It's easy in hindsight, but I've seen that happen before, and I have no doubt that if you'd considered the risk and (perhaps ironically) googled for information, you too would have known about this.
On the other side of the coin (and perhaps the reason Google haven't cared enough to fix the problem), SMTP is nicely designed so as to not result in this sort of thing losing any mail - "well behaved" email systems will just queue and retry mail for 5 days if needed.
Okay so what if the transition took five days? How about thirty days? In the absence of seemingly any warning information at all on this, how does one ever perform such a change?
Let's take it further -- what if adding a user took down your email for a month? That is just as rational as removing an artificial limit is ("Oh didn't you know? You pushed our global user database past the threshold so we had to migrate platforms"). How about if you send an email that you CC to ten people and that takes your email down for days?
If this wasn't an expected behavior, and is seemingly a mere administration change, there is no universe where it can be pinned on the user. Doubly obvious given the confused responses of customer service.
Randomly clicking things like "change my email system" buttons, then complaining afterwards that it didn't work how you expected is the sort of mistake most of us have made at least once.
Once you've suffered through those mistakes, you tend to view phrases like "expected behavior" and "seemingly a mere administration change" a lot more suspiciously.
If it's mission critical, don't "assume", don't "expect" - things are often not as they "seem". As they say "Trust, but verify." Yeah, Google (or Rackspace, or MessageLabs) will _presumably_ "get it right", but when the consequences of "presuming" are business-destroyingly-high, verify the presumptions first.