Gmail back soon for everyone | Hacker News Reader

Gmail back soon for everyone

gmailblog.blogspot.com

79 pointsmjfern50 comments

"If you were affected by this issue, it’s important to note that email sent to you between 6:00 PM PST on February 27 and 2:00 PM PST on February 28 was likely not delivered to your mailbox, and the senders would have received a notification that their messages weren’t delivered. "

That's pretty scary to me. I would've hated to have gotten any e-mail alerts or payment due date reminders during that time.

Since they got a warning that the email address was temporarily unavailable (at least that's what they should have received) any well configured mailserver should have resend the email.

From a bounce I got:

    ...

    From: Mail Delivery Subsystem <mailer-daemon@googlemail.com>
    To: [me]@gmail.com
    X-Failed-Recipients: [someone else]@gmail.com
    Subject: Delivery Status Notification (Failure)
    Message-ID: <20cf30433ea07a2745049d4fe20e@google.com>
    Date: Mon, 28 Feb 2011 04:12:14 +0000
    Content-Type: text/plain; charset=ISO-8859-1

    Delivery to the following recipient failed permanently:

         [someone else]@gmail.com

    Technical details of permanent failure: 
    Account disabled

    ----- Original message -----

    ...

Not exactly the kind of thing that would make you think resending would get the message through.

Yeah, scary would be no bounce, no delivery. Bouncing is relatively graceful.

even more graceful would be keeping track of bounces for 24 hours so you could at least tell someone what they missed if there's a cliche like this.

glitch!

How on earth do they backup all that to tape?

Must be a massive amount of tape.

If you look at http://www.oracle.com/us/products/servers-storage/storage/ta... it talks about the reduced cost of ownership for a 20PB tape library, so there are tape products available for large volumes of data. The SL8500 supports 500PB of raw storage.

If everybody filled their mailbox, 500 PB would back up 66.1 million Gmail accounts. Since they don't, we're probably looking at everybody on Earth: 73.8 MB per mailbox, which sounds like a pretty good shot at an average.

Looking at the SL8500, I think I'd intentionally lose data just so I could play with it to restore it.

>> Looking at the SL8500, I think I'd intentionally lose data just so I could play with it to restore it.

And this is why high-paid sysadmins are serious nasty hardasses when they're clocked in. :)

Would 74 MB really cut it? I'm probably "above average" but mine uses 3 gb.

There's probably a lot of people that don't "get" Gmail and delete all of their messages. There's probably also a lot of people who sign up for an account and don't use it, ever, and they contribute to the numbers. Keep in mind ~80 MB is for everyone on the planet (all nearly 7 billion) in 500 PB.

I'm betting the average is a lot lower than we'd think.

74MB seems little to me too (I am thinking of how often people have sent me ppt of cute cats, word documents of a todo list and tiff scans of some thing they drew).

It would be nice if grandparent has some reference for such a number to share

Consider de-duplication.

The hard part is backing up that kind of volume of data frequently enough that you always have a recent backup :)

To quote from The Ringworld Engineers: "The Kzinti Patriarchy is not normally terrified by sheer magnitude." I can safely say that the same statement applies to Google.

Total conjecture, but 3 ways come to mind: De-duplication, compression, and hierarchical storage management. Keep less-frequently accessed parts of the data 'Nearline', and snapshot the lower-demand, older data slightly less often. Continuous data protection appliances were probably designed and tuned specifically for this problem set, and one could assume it was built into the original architecture so that an appropriate amount of the infrastructure already had backup space in mind and accounted for.

i love how they keep mentioning "0.02%" so you think it's some insignificant number, but it's actually something like 150,000 accounts.

150,000 accounts out of SEVEN HUNDRED AND FIFTY FUCKING MILLION had trouble accessing email for LESS THAN A SINGLE DAY on A WEEKEND. No accounts or emails were deleted.

I'd argue that is an insignificant number.

you'd probably have a different attitude if it were your account.

Why? Because I'd need to wait 20 hours to get that critical life-changing email? If it was critical, someone could call my home phone, my cell phone, text me, actually have the local police stop by my apartment, etc.

This winter was pretty bad here in the US. More people (and much higher percentage of some company's user bases) lost power for longer periods of time in freezing temeratures.

More people die in a single day than the people who temporarily lost email access.

Sure, It'd suck in that period where I thought I lost all my emails, but all's well that ends well. Lets keep things in perspective. This was more of a "SHIT THE CABLE'S OUT" issue than a serious life changing problem for anyone affected.

It turns out that I was hit by this issue. And guess what - it was a big deal. For starters, it is a MONDAY here in Australia. There was no support information on this, and no help from Google.

Secondly, the login message was "Your account has been disabled". If I could log in, and there was a message that my email was being restored from backup because they had a failure - fine. That wasn't the case. They had disabled my Google Apps Administrator account (same account as my email), email was bouncing, and I couldn't log in to my email OR the administration interface. The "reset your password" and "unlock your account" links failed to unlock it.

Comparing an email outage to people's deaths is poor form. Of course they aren't the same. That doesn't make it an issue when email is a super important communication tool.

I've spilt out my admin and day-to-day accounts. I couldn't articulate why I thought it was a good idea but you've given me at least one!

So what if your admin account would be affected and your personal is not? Slightly less trouble, but still!

Sounds like a good idea. I should do that right now.

This is what free gets you!

Google Apps is not free.

There is a free Google Apps and a paid Google Apps ("for Business," formerly "Google Apps Premiere" http://www.google.com/apps/intl/en/group/index.html)

But your boarder point still stands - the OP may well be a paying customer and he did not indicate he was using the free edition.

People who were affected had their entire Google accounts disabled, and upon trying to log in, they got exactly the same messages they would have gotten if Google had decided to delete the account for ToS abuse. Additionally, since the entire account (not just GMail) had to be disabled in order to repair things, stuff like shared google calendars went offline, so users whose accounts were not directly affected were getting misleading error messages, too.

And the bounces didn't stop until the end of the working day on Monday on the east coast.

You're both right. It IS an insignificant number, but it would suck, out loud, if you happened to be in that number. (I was not.)

I have 2 opposing feelings on the matter; 1 is that email is not backup. If you're using it as such, well, you run this risk. Email is a communication medium; if ideas or information presented therein are important to you, save them in appropriate places. Email is only a way to get the idea/info TO you, not where to keep it. And yes, I realize reasonable people disagree here.

The other is that at least with Google, the account is intimately tied to any Android devices you have. I've heard reports of people not being able to use large swaths of functionality on their phones or tablets due to them being in that insignificant number. That would make the suck possibly an order of magnitude or more worse.

Not that I'm disagreeing with you, but the 750 million figure is way off. According to Wikipedia:

"As of November 2010, [gmail] had 193.3 million users monthly"

Bringing the total number of affected users down to 38,660.

Not nothing, but not a lot either. My company has an exchange server for 200 or so people, and at any one point there are probably one or two people who are waiting for a password reset, errors with Calendar, public folder permissions errors, or other human error on the part of the helpdesk interns. That's a 100x increase from Google's error rate in this scenario, but it's perfectly acceptable within the context of our organisation.

That's why we have tape backups (which can take some time / effort to restore from). I think the difference is that if a user at my company has a problem they can call a human being and be told that someone is working on it, but with Google you can't do that.

While I partially agree with you...

> No accounts or emails were deleted

This is not true. During the affected time period, those accounts could not receive email, and sent out permanent failure bounce messages. If you were one of the affected and you were sent an important email during that time... You didn't get it, and you might never get it.

It's been more than a day for me. It's Tuesday and I still have not regained access to my GMail account...

I really hope that I'm not using anything you created on the web. I would hate to be a customer of a service whose owner think 150,000 users are insignificant.

And that's bad compared to what exactly?

What alternative is online 24x7, with no downtime for backups, upgrades or power failures?

Does on-premise Exchange do better? Lotus Notes? sendmail?

Even if you assume there are 200 million active accounts, %0.02 of it makes 40.000, not 150.000

And how many of those are likely to be active and be expecting serious email instead of newsletters and spam?

I presume the same percentage as all gmail accounts. I saw nothing that indicated that these users were less active than anyone else.

Unlike Hotmail, I think a very large percentage of Gmail accounts are pretty active.

In what world does that matter?

I think the word "tape" tells you all you need to know about how serious this issue is.

The original commenter pointed out that 0.02% is a more significant figure than the Gmail team makes it out to be, and that 150,000 is very significant. I'm saying the 150,000 figure isn't as significant as it looks, either.

0.02%

"We released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email. When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version."

Some bug!!

It is worth noting that this is the first significant data loss issue that I am aware of involving gmail. As much as the incident sucks for those involved, that is a pretty good record overall.

This is great, last night I went to bed thinking "How could Google just walk away from this problem without fixing it, don't they have backups, what about corporate users?"

And this morning "we have backups, we're restoring, oh and we're trying to fix what caused it in the first place"

Would it be safe to assume then, that Google tests it's software upgrades with 0.02% of it's gmail account? (However many thousands clients are affected, it's probably better than affecting ALL their clients).

I wonder how they choose their 'test' client base.

I can't remember having this sort of problem with yahoo mail. Then again they don't build their own filesystem.

Yahoo is far worse. When they lose your mail, it's gone.

Yahoo Mail lost two of my accounts over the years. The first time was early in the service, and they seem to have lost all their accounts at once. Suddenly you couldn't log in, and there was an apology message saying that you'd need to create a new account. And that you couldn't get the same email address back, which is why my @yahoo.com address had a 2 at the end of it for several years...

Until they locked me out of it one day. As in, I checked my mail at night, then in the morning my password didn't work anymore. Since they have no human beings working for them, an entire month's worth of attempts to get access back resulted in nothing but a big pile of form letters and links to the Forgot Password page.

So no, you're mistaken. Yahoo does have this problem. If you have an account with them I'd strongly advise you to back everything up. Then switch to Gmail.

Yahoo also charges a monthly fee for access to pop3. Their web interface is slower and buggier than gmail's. There is free unlimited storage, but a smaller limit on file sizes. The virus scan on outgoing messages is usually slow; if gmail is doing virus scans, it's on the backend, it does not delay the user.

Oh, I'm on Gmail already, trust me. But thats because I like the faster interface, search capabilities, being able to check multiple mail accounts, gtalk etc. Yahoo is a great repository for my non-friends and family email (semi-spam).

I guess I've been very lucky that never happened to my Yahoo account.

Reply on news.ycombinator.com