Google Intrusion Detection Problems
fredtrotter.com
fredtrotter.com
The majority of Hacker News pundits will all have the same gut reaction to these kinds of anecdotes: complete and total lack of surprise. And although not all of us are in a position to control the vast budgets of the enterprises that drive much of this marketshare, the opinions of the rank and file do have a vast impact and I believe Google's unstable support and product commitment is one of the biggest things holding them back.
1. http://www.nytimes.com/2016/07/25/technology/google-races-to...
Real example: we use Appengine, and the other day I spotted a bug in a request handler. I filed an issue on the Appengine issue tracker, and have been having a back and forth conversation with a a very real (and very skilled) engineer about how to resolve it. This is outside our support contract, so anyone with an software issue has exactly the same access.
This doesn't cover situations such as the OP experienced, of course - but if they had a support contract, I think they'd have found the issue was resolved quickly and efficiently. We pay $150 a month for a silver support contract, and when we've needed help, we've got personal, prompt and effective support.
At the end of the day, you get what you pay for. If your organisation depends on any service, you need effective support, and you need to pay for it. That goes for Microsoft, Amazon, Salesforce or anyone else - good support costs a lot, and they all have to cover costs. Each company uses a different model, and some include more support costs in their base charges than others.
I happen to like Google's model where dipping your toe in the water costs very little, and when your service proves itself you can ramp up support as you need it. But it leaves Google open to criticism like this the OPs, who let their small project grow big without contingency planning and then complains when something goes wrong.
Finally, the only reason I'm commenting here is that I was so impressed with the support from the engineer mentioned above that I said I would mention him next time Google's perceived lack of support came up on HN. Promise kept!
Time and time again Google products prove to make the happy path happier, but as soon as anything should go wrong, you're on your own with no explanation, and no support.
Additionally, and as an aside to the core issue here, I've found the recent Google Cloud UI updates to be a real pain. When I last used them, I did acknowledge that it was beta, but I seemed to lose an awful lot of oversight of an awful lot of things. It's put me seriously off it vs. AWS.
The support page telling you to use a widget that doesn't exist is really going above and beyond in the realm of user-hostility.
When you delete an account, DB keeps charging you. There's a nice FAQ that explains to push a "review subscription" button and blah blah, but in reality that button doesn't exist.
The button isn't there.
Support will "fix"it for you one-off, but not change the button. Wonder if this dark pattern is worth 3-4% of revenue.
Maybe they expect you to cancel the card upon deleting an account.
It can be said with certainty that this cannot be an oversight. Not many look at their credit card statement anyways.
It can be, especially when you're using something like Stripe. It only takes one failed call without retries (or a period of time where networking isn't working quite right) for an account to be orphaned.
Arguably, those sorts of calls should be durably queued and retried until success, but it often doesn't happen.
PM decides it's cheaper to have support field the one-off requests than to spend a few days making those changes.
Later on, another developer accidentally disables the call and there isn't sufficient test coverage for disabling billing on account removal.
It's a story that'll be repeated over and over.
https://www.dropboxforum.com/hc/en-us/community/posts/201886...
Can't really compare, but AWS isn't great either.
I've experienced instances when an instance (1) shut down, (2) rebooted on it's own.
AWS support would point you to the SLA if you were to report the incident.
Separately, even hardware you own and operate is ephemeral. You never know when the disk, PSU, or cooling fan will kick the bucket and the host wedges one way or another.
If you want the illusion of permanence, VMWare had a solution for you ... for a price you're not gonna like!
Everyone should have backups and a plan in place to recover when a node goes offline, but this isn't exclusive to AWS or cloud in general. A robust infrastructure will have failovers standing by and accept the possibility that any one machine could die at any moment. Shouldn't have any single points of failure, as colocated hardware kicks the bucket unexpectedly sometimes too.
Well, no, its not, otherwise "private cloud" would be incoherent.
_it_ in this case refers to the ephemeral state of EC2 instances. You literally did suggest they were ephemeral.
Best, Terrance
It seems like you guys really need to get some call centers filled with people with some level of diagnostic ability and problem-solving authority. Yes, this is expensive and difficult to get built, but if you really want to Reduce Customer Anxiety, nothing can do that like the knowledge that there's a real person available 24/7 who can fix any problems on Google's side. Make it pay per incident or by subscription if needed to keep the jokers out, but anything is a whole lot better than nothing.
"Reduce", not "remove" - it's stinks of implying there's an acceptable level of "customer anxiety" that their platform should generate, and someone's set some easily-measurable metrics and a bunch of people's KPIs - and those people are now all working out how to game the metrics instead of solve the problems. "Hey, my bonus depends on reducing the clicks on the "Request an appeal" button. What's the easiest way to meet that goal? I _could_ start handling customer service in a professional and sympathetic way - but that's not "The Google Way". I know, I'll just move the button to a different place this quarter, rename it next quarter, and remove it all together before the end of the year."
Sorry Google-people - I know you all don't think you're doing evil, but there'a a hell of a reputation you got to overcome on the "Google just doesn't do customer service" issue.
That sounds entirely too plausible as the real root cause of this problem.
(I also get 'man bites dog' will also get more traction than the inverse, but regardless - this kind of complaint is a very common one about Google's products, from consumers and businesses alike).
I'm looking at the postmortem now and without wishing to jump the gun and talk about things I can't, it looks like this is being taken very seriously and a number of improvements and bug fixes are going to result. In this instance, I think it's doing the Cloud Support people a disservice to call them "disingenuous".
Disclaimer: Used to work on Google Cloud, now on Google Open Source Programs Office.
> I think it's doing the Cloud Support people a disservice to call them "disingenuous"
The problem is that to us from the outside, they do look the same.
In fact, that email in the article, and the subsequent events, are in exactly the same style as the "ban hammer" Play Store publishers get. You really can't blame us (Google's actual paying customers) from equating one with the other.
(And no, I do not equate consumer Windows support with enterprise Windows support either, and as it turns out they are not the same level.)
Given the story so far, you offer adwords style customer service (we don't give a damn about you) while attempting to catch up from position 3 behind aws and azure in the paas race. Complete with automated emails with outdated documentation and no human anywhere without a social media outcry. I'm sure stories like this are damaging (seriously, just use aws), and there will be some postmortem damage control, but unless I see a human support sla for situations like this, I can't see why anyone would risk their business on you.
Seriously, a fix it or we shut you down email for reasons you won't share with an 2 business day response. Of course, you're issuing a 3 day warning, so if that fell on a Friday, you would be dark before anyone bothered to look at the support ticket (3 days expire Monday, 2 biz days Tuesday evening). Amazing.
That said, we still use Google Cloud for some part of our infrastructure and enjoy its technical side.
[1] The billing credit card for one of our projects expired weeks before any invoicing would happen. Instead of informing us —after all we were paying customers for many months before, there should be at least some good faith— all our projects were disabled many minutes (maybe 15 or 30) before we got a confusing email about the issue at hand and the total downtime was two days.
Reminds me of the emergency telephone signs on the Golden Gate Bridge. They once read "Emergency Telephone in case of Breakdown", so people could call a tow truck. Then they were changed to "Emergency Telephone for Psychological Counseling" and now connect to suicide prevention. I can see someone with a flat tire on the phone, "And how does having a flat tire make you feel?"
This mindset is totally inappropriate for a B2B service. The caller is probably not anxious. They're at work, doing their job, keeping something working. They need repairs, not grief counseling.
But it's just marketing, at the end of the day support is still a large expense that some companies believe they don't have to bear. This seems to work for some companies better than others, likely due to different business models leading to different customer expectations and requirements.
It will take way more to show you guys really are reviewing similar incidents and, more importantly, to actually act on your reviews and improve your processes.
I wrote a little app to help myself learn birdsong. You copy some mp3s to a folder on your sdcard, open the folder in the app, and it will shuffle them and only show you the name of the file if you ask. Pretty simple, but I found it useful and thought it might be useful to other people. So I pay my $25, upload my app, and wake up the next day to an email saying my app had been suspended for "deceitful behaviour" and that it happened again they might suspend my other Google services, for instance the gmail account I've had since 2004.
Of course the email doesn't contain any information on what the deceitful behavior was, so like a Kafka story, I'm stuck defending myself against charges that I'm never informed of. When I heard back from support, they say that my app opens to a list of mp3 apps to download. So either my account has been hacked and someone changed the app apk, or they are unable to tell the difference between a directory listing from the sdcard and a list of mp3 apps. If the latter, how my app can be deceitful when it matches one of the screenshots I attached is completely beyond me.
Once my email migration to fastmail is complete, I'll rename my app and try again, this time with a modified UI that makes you click a button before showing a directory listing. I'd like to get my $25 worth if nothing else.
This whole incident has left me with zero confidence in their customer service or technical competence.
Buying a house built in 1910 and having to remodel and move to it financially prompted me to buy a smart phone, and Project Fi was clearly the best match, so I've now seriously reversed that ... but that also means I now won't even think about increasing my exposure to these sorts of risks by trying to write apps for the Play store, using any of their cloud offerings, etc. My main phone and its number are too valuable, too sticky to risk.
* locked out of your account
-Terrance
I'm not saying that's actually what is happening, but it feels like it to me.
People want their accounts reactivated.
Tscanausa @ Google
1. It's absurd that an automated process accuses you of trying to intrusions of third parties without providing details of what has been detected. This leaves the "accused party" (your paying customer) without ANY idea what is wrong or how to fix it.
2. The automatic suspension of the entire project within 3 days is similarly out or proportion, especially without Google attempting actual human contact first (a phone call, for example).
3. The appeal-process is apparently literally broken. Even if this weren't the case, disabling a customers entire project in an automated fashion requires WAY more ways of directly reaching Google to prevent this. Give a direct phone number in "emergencies" like this. We know it costs money and training. We expect you to accept this as a cost of doing business.
The above leads to the (at the moment rightful) impression that Google Cloud isn't ready for business, especially not small business. Turning off what could be someone's livelihood is a testament to the (possibly unintentional) arrogance that is widely perceived from Google.
There's very little information and/or reassurances.
I have learned that it is way too risky relying on Google services for crucial business operations.
If anything, this is one of the more generous stories, because Google gave them a day or two of notice first. Usually you just find your Google account shut down one day.
But in some instances it might be financially worthwhile to take advantage of their services (particularly their free or very cheap ones) as long as you have a plan for standing up something else when that happens.
For instance, I'm pretty comfortable using Gmail, even though I know they might someday pull the plug on my account for no particular reason. I'm comfortable doing that because all my mail is backed-up elsewhere, and in a pinch I could restore everything or stand up a new mailserver in a day. It would be obnoxious, sure, but it's not exactly breaking new technical ground or anything. The years of free service I've gotten out of them make that risk manageable.
I could easily see a company taking that bargain for various other services. The problem becomes when you simply rely on a 3rd-party's infrastructure as if it was your own.
The number of people that can be served at very little cost without any human involvement is apparently very lucrative. Google just considers customer service as an archaic relic that predates the invention of behavioral algorithms.
Also it could be that customer support is just overrated.... The problem is that 95% of support calls / issues are anyway just people complaining about things which are not at all related to the actual Google service.
But, on the other hand, hackers which try Google Cloud and are bitten by lack of Google support do have influence on decision making process in big corporations.
Sometimes the end result is less important than how you get there, and how you treat the ones who don't fit into your plans. It happens all the time with big companies, but Google even more than most. They KNOW that the potential for problems exists, but so long as it only affects 5%, who cares? Fuck you, got mine.
I think that the attitude conveyed by Google's lack of support is what inspires a lot of people to post rants online, rather than the frequency of incidents.
Then you consider this when you decide whether to base the next big business decision on an AWS or Google Cloud platform…
About once a year, there's a serious intrusion complaint, as the crawler, which obeys robots.txt, examines about 20 pages on a site in a few seconds. No more than three connections at once, but some sites are touchy. The server leasing company sends me a warning letter, I reply and call tech support, and there's no big problem.
I can recommend leasing servers from Codero as an alternative to dealing with the Borg of Mountain View. I've been a customer for five years, and nothing bad has happened. They now have "cloud services" too, but I haven't used them.
That seems like something that could be slowed down slightly with little negative impact, and a big positive impact (no annoyed people, no interaction with support).
http://www.businessinsider.com/google-cloud-won-skeptic-afte...
Google then shocked him again, in a good way. Within four hours of tweeting, someone from Google had contacted him and had restored access to his project.
Trotter says that, it turns out, he and his team bear some responsibility. They had inadvertently set up a server wrong, exposing a hole, and a hacker was using his company's to conduct a "denial of service attack," which is when hackers overload another website or online service with so much traffic, it shuts down.
Sorry to be so harsh, but Google has always been this way.
Why do people keep putting essential stuff in someone else's sandbox? Your effectivly adding a SPoF, that you have no control over
I've had similar problems in the past with Amazon shutting down services and providing little to no support to get it remediated. Even after paying for technical support, it was impossible to recover (in our case, it was a billing issue that was entirely Amazon's fault).
My general recommendations are:
- be sure you have fully scripted your deployment process, - be sure you are making remote backups of critical data (often times this is as easy as setting up simple replication of S3 to GCS or Azure's Cloud Storage) - rely as little as possible on their "value added" features - if you do rely on their value added features, be sure you still have someone on staff who can quickly replace it with something minimal (e.g. know how to install your own mysql)
At the very least, this will put you in a position where you aren't 100% locked out of your data, or have to completely rewrite your application in order to move to another provider.
You don't have to go crazy with some kind of hot failover, or active-active deployment. Just have daily snapshots of your data and mitigate the risk of your PaaS provider.
No matter who your hosting provider is, you should probably have this sketched out as part of a "disaster recovery" plan.
Nope, and thats why I don't use them. There are situations where they make sense, but people should be much more skeptical about them. I don't have stats, but IME I've yet to see a company that actually saves money by using them. I've seen one company that did save money the first year, but got too addicted to "just spinning another instance up" and stopped optimizing their code. The next year they were bleeding cash.
So, speaking from my experience: It's more expensive, less configurable, and unreliable. Why do people keep buying the cloud lie?
Now it's obvious that companies just want to save money, but another reality is that those human processes just cannot compete on the service they offer to their customers, developers. You could have the best QA team in the world, but compared to a dev team armed with mature testing practices and CI, it's seconds vs. hours for developers to get the feedback they need in order to ship.
Infrastructure hardware is cheaper than ever these days in order to compete with the Cloud, while good developers are at an all time premium. When infrastructure is no longer a slow, scarce resource, it enables developers to spend more time perfecting their applications and experimenting with new approaches. So I suggest you not approach the Cloud as purely a cost saving measure for the infrastructure department, and really think about how transformative it can be in changing your IT processes.
Having spent years help running a physical hosting company, working with virtualized servers saves LOTS of time. I still find it amazing and much, much cheaper in staff-hours to reboot a cluster and have it come back up on 2x larger hardware when a project grows.
And that doesn't even touch the trust you have to have in your provider.
And I love virtual more than physical. But I set up my own virtual server and run them myself.
Where do you keep the machines? In your bedroom?
For work, we have a DC. For home, yes, I keep them in my office.
So your strategy for scaling up is building a data center then?
Not really. Just a normal home connection. If you design your services sanely, and don't have much traffic, it's not really an issue.
> So your strategy for scaling up is building a data center then?
Well, it depends. You really should do a cost-benefit analysis, because each product is different. But if it makes sense, then yes.
Well, with everything, there are pros and cons. My argument is that I think it's silly to trust someone else with your infrastructure. There can be benefits to doing so, however, your giving up a lot as well. And in terms of cost (which is usually the main argument that I hear), I've personally never seen it work.
Your trusting someone with something pretty critical with your business, so you should think long and hard about which direction you want to go. If your cloud-service gets arbitrarily shut down or has an outage, what responses can you take to fix it? Usually not many. Thats essentially adding a SPoF, which is generally considered a big no-no, although it does happen.
> The kind of things I was doing half a year ago you couldn't run on your home connection.
I'd be curious as to what your doing then.
> just a signal that you're not doing anything great.
I guess that depends on what you mean as great. I do a lot of useful stuff: email, web-apps, file-store, home-automation, etc.
Likewise, I really don't want to pay a power engineer to troubleshoot why our automatic transfer switch failed to transfer after a power failure and it took down our entire datacenter, I'd rather let Amazon deal with that while I failover to my backup servers in another AZ or region.
And my 100mbit ethernet handoff with a DS3 backup doesn't give me much clout with the teleco when the network is down, but AWS's multiple OC-192's (?) gives them a lot more leverage and more redundancy.
Owning your own infrastructure doens't mean that you don't have to deal with a third party provider, but it does ensure that you're caught in the middle of it more often.
And I have far more control than I had with colocated equipment, if I find I need to scale up my services by 2X, 5X or 10X, all it takes is a little more money during the time I need the extra hardware (i.e. end of month processing, plus holiday shopping season), I don't need to order hardware a month or two in advance and pay for it all even when I don't need it.
What I used to pay for my network infrastructure alone (hardware + support + network engineers) pays for most of my entire AWS compute infrastructure.
In my last work place we had paid AWS customer service, and they were outstanding.
But before I got my first HostGator account, I called their tech support line at 2 AM, and got a guy named Chris with a strong Texan accent. And I've had escalated issues at odd hours where I've actually spoken directly to sysadmins there. Ticket support only hosting simply can't compete with that service, no matter the cost.
> Google sucks
> Don't rely on Google if you want your business to succeed
> This is old news! Happens all the time
> Google = SPoF
...you're all looking like trolls to me. Please, do tell how Spotify manages to service 60+ markets with some core pieces of infrastructure on Google's Cloud offerings.
I've had this issue with google too. They'll shut down your account, your whole account - google apps email and all - if an automated system detects something dodgy. In my case an automated system at ebay accused my site of phishing which it wasn't.
There is a paid support option, but when your account is shut down you are unable to access the required code.
I don't think you can appreciate how inaccessible Google is until they decide you're a bad actor.
I also imagine that Spotify pays a lot of money and has a dedicated account manager.
The same thing happened to us in 2013 luckily had backup servers at Linode.. took them 2 weeks to solve the problem it was a bug in their billing system.
Google gave us $8k credit and I gave them another try and convinced myself that GCE was very young in 2013.
Now I'm just worried! It's 2016 and our entire business depends on their platform and their support sucks! Even the $400/month version!!
Google, why can't you have support like DigitalOcean and Linode?
Because it's totally not in their DNA?
Because GCE and company are relatively low priority offerings? Would you agree these are higher:
Ad Words etc., which make the bulk of their money.
Search etc. which among other things has you seeing those ads.
Android and all their other efforts to keep themselves from losing their access to the users of the above.
For a while, Google+ was an anomalous offering they were pushing very hard (with this sort of brain damage with linked accounts that hit hard with their real names policy), but they might try such a stunt again.
Compared to the two service providers your mentioned, which only do one thing. Even AWS is I gather rather like that, there's it + all the Amazon selling stuff which is significantly separate. Some of the DNA is shared, but that's not all bad, such as the focus on customers.
Also, the hypocrisy in using very logical questions to hire people but not sticking to similar logic in the product line is laughable.
It's more like they have an entire company of Sheldon Coopers, without a single Penny or Leonard Hofstadter to round things out.
http://newsfeed.time.com/2013/12/10/meet-the-robot-telemarke...
----
I really don't understand how anyone can use Google for production. If you don't pay for support to have a way to actually get a hold of Google...
... Someday you may be in a world of hurt and have absolutely no way to get ahold of anyone. You can post a story and hope and pray that Google might read it and have someone reach out and remedy the situation.
However, despite using our credit to buy gold support when it came to using it I couldn't work their tickets UI at all. Turns out you need to change products from your Google apps support to cloud platform by some tiny link which is not at all obvious.
I had to phone up and it was obvious people hit it all the time as the operator instantly knew what the issue was.
Also related: azure by default didnt renew your free trial despite an active cc being on file. It just shuts everything off with no obvious warning. What other service does that?
Honestly think these companies need to do some proper usability testing on their flows because there is so much clunky weirdness in cloud providers right now.
"Google offers support solutions where you can talk to a person if you have a problem. We view it as problematic that interrupting an “allergic reaction” as a “support issue”. However, we would be willing to purchase top-tier support in order to get this resolved quickly. But there does not appear to be an option to purchase access to a human to get this resolved. Apparently, we should have thought about that before our project was suspended."
I think he means "there is no 'pay to talk to a human'" option in the FAQ/etc for this issue.
That is, he wants to pay to talk to a human about just this issue, and thinks somehow that paying for the support option will not give him that (or is too expensive for that).
I almost can't believe that Google doesn't have a support line for its account holders for Google Cloud, when you have companies like Paypal and United Airlines, which have many more users (paying or not) and have support lines.