Google Services Experiencing Disruptions
google.com
google.com
I've suggested to our team they copy/paste locally anything they're currently editing.
"Sometimes, it's probably almost approximately around what we think could be along the spectrum of known values." - Me, as CEO.
You need to draw a line in the sand somewhere, but whatever measure you choose is going to be somewhat arbitrary so I think it's reasonable to have a human make the final call (based on some known criteria).
Then put that on the dashboard! It certainly is an outage for the affected users (by your scenario).
- 999/1000 Data centers up
- 100,000 Users without service
- 6 paying customers without service (will get money back)It is frustrating that these pages are not updated in real-time, but I do understand wanting to be sure before publishing a message to your entire userbase that you are experiencing disruptions.
Gee, you mean we just discovered that area where Google admits that either automation or machine learning does not actually work?!
Google doesn't need an internal party to hold responsible for the concerns of free mail users or their own contractors who are called "content creators." In part because Gmail users and YouTubers don't lead fortune 50 companies and play golf with Google Execs like Google Cloud customers do.
https://mobile.twitter.com/uhoelzle/status/13093135569956618...
So, "kind of" confirmed, I think. Assuming a bad update is the most likely culprit for a whole pool of servers to all crash at the same time.
To get ahead of the cynics — it would not serve the least generous of Google's objectives to be less than transparent about downtime — people figuring it out while the dashboard is green looks much worse.
A lot of customers judge their perception of the severeness of an issue based on the service dashboard. That's why the dashboard makes everything look more rosy than reality.
Every time it says "some customers may be experiencing elevated latency", it usually means "all customers are seeing all requests timeout, and the only ones who aren't are the ones not using the service right now".
Yes seriously - when they say "<5% of users are having trouble accessing Gmail" they are calculating that based on the percentage of all Gmail accounts which have seen an error. Obviously the vast majority of accounts are inactive at any one time, so aren't seeing anything...
Also SLAs and service credits are tied to these official notices which causes even more delay before status updates are approved.
I think I'd prefer a status page that reflected that, or even no status page at all with an explanation, rather than a lying one, though.
The on-caller for the specific service decides on how to produce.
The status dashboard is something which will be (manually, yes by hand) updated by an operations employee, who is a couple of layers behind the actual SWE/SRE who gets the page.
It's complicated, and the human factor is massive in an unpredicted scenario.
I'm an IM myself, and regularly have to make the call regarding status updates, and also seen how other companies do it.
One of my favourites was a big it infra company sending a notification about an outage 6 hours after the event, after we opened and resolved a ticket with them. Was actually pretty useful but made me laugh all the same
Although I'm not sure if this is what stack exchange actually uses or if there's some private API for big partners.
For example, if you type a search in the homepage or address bar, you start getting suggestions instantaneously. Think about this... if Google decided to only give you suggestions every second keypress, or every third keypress, would you really notice?
There are two mechanisms helping keep search up when considering ads:
-> Search engine pages can be served w/ no ads, or even no personalization if components are down or lagging
-> Ad logs can be lost if necessary, since we'd under-bill in that case, which would be our own problem. That happened a small handful of times while I was there.
The billing system is indeed transactional, but has less need for 100% up-time and 0 latency in comparison to search. Not nearly as big of a deal if there's an hour delay on billing ads as compared to search being down for an hour.
It annoys me a lot that Google doesn't use the "real" URL in their search result links.
The good news is that it happened on a Friday -- see you all next week!
The apis tested by the automated system could all still be working. Probably the only way would be to have a constantly running vm testing everything via the web ui.
Edit: YouTube livestream is up, despite downdetector claim that it is down.
Docs, inside and outside of GSuite appears to be up.
1836 Pacific: Downdetector shows improvement across Google services.
1840 Pacific: Downdetector shows marked improvement across Google services. They don't have units, but ~80% improvement in many services over peak down-ness.
1852 Pacific: Continued improvement.
1854 going for a walk. Looks like things are generally headed for normality.
https://status.cloud.google.com/ https://www.google.com/appsstatus#hl=en&v=status
All my mail sync is down, Analytics isn't loading, Sheets and Docs are all stuck in "Reconnecting". I wonder what could cause this.
"a pool of servers that route traffic to application backends crashed"
Luckily they’ll fix the issue within 24h, which is when TPUs preempt. :)
I manage an enterprise machine learning team and we do tons of stuff on GCP. I’m dead serious: there is not a single use case where it makes sense to choose TPUs unless the main value you’re seeking is just the entertainment factor of using Google’s new thing.
Since your post didn't actually include any details, I did a search and immediately found an article[1] where TPUs worked better for their particular use-case. I suspect I could find many such reports (and probably some opposite reports too).
It's unfortunate they didn't work for you, perhaps you should give them another shot with a different model. I'd recommend using Cloud's examples as a starting point.
[1] https://medium.com/bigdatarepublic/cost-comparison-of-deep-l...
When you use them right, a TPUv3-8 gets equivalent perf to a cluster of 8 V100s.
I was astounded. I trained StyleGAN 2 from scratch at 1024x1024 in 2.5 days. nvidia took 7 days for their official model. Granted, I used a v3-32, not a v3-8, but performance seems pretty similar.
(This is probably a testament to who pays the bills.)
(I work on Compute Engine's virtualization infrastructure)
Yeah, you won't. Stop being hyperbolic. You're amazingly served by google and you'll forget this 20 minutes after service is restored.
It makes me wonder, what duration of outage is necessary for people to actually switch?
That lasted 2 weeks. Being locked out of or requiring special considerations for the first calendar invite, or video conference, or collaborative document was a complete unnecessary pain. “But you can make a google account with your external email” barf.
Its sad because Zoho is trying so hard too. Imagine planning and working on all those equivalent products. lol
Speak like a human for fuck's sake.
We are very sorry that the spam filter is giving you such trouble. It has self-learning capabilities, but algorithms are not always perfect. If you could send us your account details and the headers for some of the emails, we will look into this right away. Please send us an email at support (at) zohomail (dot) com when you have a few minutes and we will dig into the details.
(1) Look at other responses from people directly responding to issues from customers here at HN. Look at how specific, authentic, and action-oriented those responses are. Then look at yours again.
(2) Notice how they don't use the royal "We." It's 2020, we're all working from home with small spots of clean behind us for zoom, the royal We has to die. Unless the speaker is actually a committee or Borg, please use I.
(3) "not always perfect" is garbage. It failed the user at its basic job. Say so. Any apology without admission isn't an apology. Actually, kill the entire sentence. Nobody cares about the capabilities of a system that doesn't work right.
(4) They've already said that they tried you and you failed. Explicitly ask for a chance to make it right. Why would they bother working with you again, if you don't even ask them? They'd be doing you a favor. Ask for the favor.
(5) Nit: The last two sentences redundantly ask for the customer to contact you. Combine into a single request for an email with (btw, you don't have to explicitly "send us an email" when you're giving them an email address to contact -- it's a side-ways way of talking down to the customer's intelligence) account "details" (Which details? The account name? Credentials? What?) and headers.
As for the royal we...I think you are right. In 2020, with the world the way it is now, I think everyone can use more personalization and less of the anonymity provided by we. I take your point, and will do my best to implement it.
Again, thanks for your time, and for your passion. Have a great one!
One last bit of unsolicited advice: your job is writing. This is the best writing book I've ever read that deals with the issues I saw in your writing: https://www.amazon.com/Writing-Well-Classic-Guide-Nonfiction...
You can tell the author knows what they're talking about because the book is a really easy read!
Quite the opposite, it's still much harder to not use Excel if you exchange data with a lot of business partners.
yeah I'm aware. if you have a choice on a new organization, low budget but still able to access good, fast and cheap, use Gsuite.
Nobody who can afford to pay for a reputable webmail provider should be on Gmail. If not for privacy, because its design is terrible.
If you use Google Chrome, or Youtube, or Google Docs, or Google DNS, or an Android device, you should not use Google Search. That's too much data to hand to one company. I use DDG, and when it's not good enough, I use Bing (and if I used Windows, I'd rule Bing out, too).
Aside from privacy considerations, Google search quality has been declining for at least a year. I want to skate to where the puck is going, not where to where the puck is a mountain of ads followed by nonsense.
"I left Google" is going to be the 20's version of "I left Facebook."
Over the next decade, downloading and storing and processing such dumps of leaked data will continue to become quicker, and require smaller devices.
Over the next decade, mobile devices will shrink, resulting in smart glasses much more powerful than today's smart phones.
Put that all together...
It's 2030 and you're walking down the street. You pass by a stranger wearing Apple Glasses.
The stranger once spent an hour installing an "Intel on Everybody!" app, and downloaded 100 TB of random leaked data from a torrent site on everyone in the world.
As you enter the stranger's view, the glasses use the leaked data to facially identify you.
Now that the glasses know who you are, they automatically alert the wearer of the types of data they know about you.
The bored stranger occupies himself by reading your gmails from 2022, perusing compromising photos you saved, checking out your medical records, reading arguments you had here in the comments of HN, etc etc.
Maybe my understanding of technology and security is wrong somehow, but that future seems awfully plausible to me.
If the downside to minimizing my footprint is that I use DDG instead of Google, the upside is worth it.
As outlandish as my comment may come off, all the pieces (maybe aside from Google leaking data) leading to such a future are conventional takes. As far as big companies leaking data, it's not that it happens every day, so much as it is the cumulative nature of leaks: each leak adds to the pile.
I don't know the likelihood of this all going dystopian, but I did realise a while back that it isn't hard to pick a not-google option early when a thing is new. Rather than attempting to go cold-turkey you can just stop growing more locked in. After a few years you notice you're not stuck any more.
- technology is getting more powerful and smaller
- large companies can be hacked
- leaked data is easily available on the net
- smart glasses will become more common
- apps can be made to facially id people using leaked data
Are any of those statements outlandish?
websites that host the leaked celebrity nudes are torn down constantly. I can't imagine that you can upload that chunk of data or that app onto any platform without repercussions. On top of that, my tin-foil hat take is that internet access will eventually be completely de-anonymized. Social security or gov. ID to sign on.
It becomes much harder then to spread TB of personal information. Especially since even in 2035 I can't imagine there being very many hundred(S) of terabyte datasets being passed around social circles.
websites that host the leaked celebrity nudes are torn down constantly
internet access will eventually be completely de-anonymized. Social security or gov. ID to sign on.
The only realistic attack on Google would be some targeted/filtered dump of celebrities or perhaps entire GSuite organizations' emails and files - and even then it would only make headlines, not cause any real trouble for Google outside of <1% of their regular user base ditching for protonmail or what have you.
I find it hard to imagine Google being leaked
The sheer amount of data
my first computer had a two Megabyte Harddisk - around twenty years later my Phone has a 256 Gigabyte MicroSD card
it's going to happen
My take: it won't be something everybody does (unless provided by some service, which looks illegal at least under GDPR) but be prepared for extensive profiling by whoever really cares about doing it.
It also is possible someone will force Apple to permit third parties to run app stores.
I should point out that while I chose Apple in my comment, there will be other brands with fewer restrictions.
I think that there will be a new form of entertainment which will basically generate a soap opera-like experience from real lives based on aggregated camera data, recorded conversations, and the blanks filled in by guessing.
I've adapted a different strategy from you, however. I have accepted the new privacy terrain, and I lead my life as if someone is always watching, which means I try to be my best self at all times. :)
It is true that, as you point out, the majority of Google users probably prefer a less literal algorithm. Do the ways Google Search is changing benefit people who aren't me?
One way that Search is changing does align with what the average user wants: it increasingly interprets results semantically (synonyms,contextual data,domain-specific knowledge,etc) instead of just 'grepping' text content.
The other way Search is changing is user-hostile. Google is an established monopoly now, whose founders have ridden off into the sunset. As its competition and idealism have waned, so has its willingness to put search quality above profits and strategy.
It does not serve the user to include more paid content, to find excuses to personalize search, to cross-promote other Google products, to make Privacy settings inscrutable, to steer customers to AMP pages, and so on. Google has "actually, it's a feature!" rationalizations for these changes, I am sure. No matter!
If the only metric Google cared about were customer satisfaction, I would agree that their surveys, statistics, and A/B tests matter. Sadly, Google seems increasingly concerned with attributes that are at odds with their users: advertising revenue, marketing, etc.
In my judgment, even for normal users, Google Search's bad changes already outweigh the good. I reckon the bad incentives will remain in effect for years to come, too, which does not bode well for Google users.
So you mean it will be all the rage to talk about in tech forums but somehow the number of active users will continue to go up every quarter?
Bit annoying to have your email, documents, cloud services, analytics, and video conferencing all go out at the same time, especially when everyone is working from home.
I was supposed to have a Meet call with someone from the UK this morning, I couldn't join the call and they couldn't email me (I could still email them though, as I don't use G Suite for my email).
It's easier to switch than you think.
If Google's security measures had an outage ...
If our VPN was down a week we might not switch because we have an alternate workflow, but if our production servers were down for a few hours we'd switch.
Also, I missed one really important email and that's all it took to switch the hosting for that.
And now I guess I am not using Google Keep anymore, which is sucky but probably good for me in the long run.
The dashboard neglects to mention Google Keep experienced a disruption too.
This was a test of their kill-switch.
Edit: Gmail Down
(In Google's own browser, too.)
https://www.google.com/appsstatus
https://www.google.com/appsstatus#hl=en&v=status
Pretty annoying.
(If you paste the 2nd URL directly, you will not have problems.)
If you keep going to the site over and over, then potentially it counts as new views (albeit not unique ones)
That said Google probably exempts their own sites from page ranks since they you know, own the search engine
I love the history API when it's used approprately, but it is a double-edged sword that has to be used carefully. News sites injecting their homepage in history on an article sometimes make me wonder if it's worth it.
I don't bother clicking on TechCrunch articles for this reason.
If anything they'd probably want you not visiting it more.
> News sites injecting their homepage in history on an article sometimes make me wonder if it's worth it.
They even bring up a news site as an example.
Hopefully the answer isn't another permissions request :)
We really need a "update uri without side-effects, at least in #scope.