Hacker News Reader

Comment by tedivm | Hacker News Reader

People really need to understand that Google is a garbage company that can't be relied on for anything. I've spent years as a consultant, and several more in the startup world, and I'd rather host all my content in us-east-1 than rely on google.

* I've had customers get their cloud accounts shut down for a literal three cent billing error,

* I've seen people who use Google for advertising pay thousands of dollars and suddenly stop getting their ads shown,

* I personally ran a website with Google Ads, and the first time I tried to cash out was accused of click fraud and had all the earned money stolen,

* I've had product managers for GCP lecture me that I shouldn't be using "preview" services (despite services being in preview for years), and then tell my I should have used a different service . . . that was also in preview.

One of my more recent startups was actually funded by a Google based venture firm, and they wanted to know why we refused to use GCP. My answer is that I wanted the startup to actually succeed, and it wasn't worth the risk of dealing with Google's horrible support.

View on HN

> People really need to understand that Google is a garbage company that can't be relied on for anything.

Normally the response to this is, yes, it sucks, but you're not paying Google to host your data.

However, it looks like this journalist did pay Google to host his data. So this immediately puts it in a worse category.

> Normally the response to this is, yes, it sucks, but you're not paying Google to host your data.

You are not paying the competition either; so you somewhat paying Google. It's not a direct payment but indirect support

> You are not paying the competition either

Competition for what? Here we're talking about data hosting. There are plenty of companies that will host huge amounts of data for reasonable prices. And unlike Google, they actually have reasonable terms of service and customer support.

And you are not paying them, not even the reasonable price, because Google flips giving it away for "free" - so the competition doesn't get any money and Google doesn't have any liability whatsoever... Win-win for Google

That whole "you're not paying"'thing is really a straw man at this point too. When Google was young and eager to be a good netizen I'd have agreed with that and said it was all part of helping to make the web better.

But they've been entrenched for years now, completely dominate so many aspects of the web and get plenty of value out of even their free users.

Given their size and stranglehold on just about everything, pulling an "oops, sorry, I guess you get what you pay for" is just ludicrous at this point.

And sorry if it seems like I'm arguing directly with you, that's not my intention. But I see this a lot and have gone from saying it myself to vehemently disagreeing with it over the last decade or so.

Of course Google gets value out of free users--that's a huge piece of their business model. And of course if Google had actually kept to their original "don't be evil" model, they wouldn't routinely screw over users the way they do.

However, as you note, they have been entrenched for years now, and it has been obvious for that same amount of time that they have long since dropped "don't be evil". So while it certainly sucks that they do the things they do, acting surprised when it happens is not a reasonable position at this point. Anyone who expects Google to honor any kind of commitment is simply asking for trouble. I don't trust them for anything.

And if widespread realization of that fact gets people to stop using Google services, so much the better: maybe that would actually get their attention.

The average non-tech savvy person doesn't really have an option than to get screwed over then. Assuming you have an Android phone, it's practically required to have a Google account; many apps just flat out do not work without Google Play Services or without signing in with your Gmail account.

Yes, I know about microG, Graphene, F-Droid and the like, but the average person is not going to flash their phone. They will inevitably hit a wall that says "you need to login with Google," create a Google account, and won't ask questions.

Just as an example, my mother-in-law couldn't find her contacts when she bought a new phone. Why? Because she was unknowingly saving her contacts to a Google account she forgot she made.

> Assuming you have an Android phone, it's practically required to have a Google account

To the extent this is true, it's not just Google, it's also the phone manufacturers and service providers (like Verizon).

You can use apps from F-Droid without flashing your phone. The only apps I have from the playstore are apps for a few financial institutions, otherwise it's all F-Droid.

And you don't need to be hackerman to do this, I told my retired parents "you should check out F-Droid" and they did. Now they recommend F-Droid apps to me.

But you and I (and perhaps your retired parents) are the exception. Most people want to have their bank app on their phone. They can't get that on F-Droid, so they go to Google Play Store.

Last I checked a bank is a financial institution, which the GP said they still get from Google Play. But that doesn't mean you have to get all your apps from Google Play, which was the GP's point.

> But they've been entrenched for years now, completely dominate so many aspects of the web and get plenty of value out of even their free users.

That's why I store only 1 file on my Google Drive - a 10 GB encrypted blob (TrueCrypt partition).

Why not VeraCrypt, honest question?

I looked into that myself years ago, and honestly can't remember now why I made that decision back then :) Should probably document such stuff somewhere.

Good point. I have never considered this argument before, but I agree. In a healthy case, the sphere of responsibility (of a company, individual) should approximately match their sphere influence. And with a sphere of influence this large, this goes far beyond the "contract" between individual parties only.

In some cases this kind of relationship auto-regulates itself (e.g., taking a small vs large credit in a bank): at larger scale, both parties are invested into maintaining a good relationship. The problem in this case is, any individual investment into any relationship with Google is just a drop in the ocean.

Google just spends millions of dollars building a product then dumps its own baby on the streets.

Then goes and makes new babies.

>but you're not paying Google to host your data.

ok but a farmer that throws their cows down a ditch is arguably much worse than one that milks them.

It's always been a garage argument. If someone is offering free food, and it gives you food poisoning, on purpose, you don't say "well it was free".

> If someone is offering free food, and it gives you food poisoning

That's not a suitable analogy for users of Google's free services. A suitable analogy would be, someone is offering free food, but (a) its quality sucks, and (b) they arbitrarily withdraw it from a significant number of people for no good reason. Yes, that's unpleasant, but "well it was free" is a suitable response in that case.

The issue in this particular case, though, is that the journalist was using paid Google services--which is a very different situation.

> but you're not paying Google to host your data.

Funny, because they never seen to have problems hosting the advertising/surveillance data they collect on me. They never went "sorry we can't track you any more for free, will you type in you card number so we can keep surveilling you?"

They don’t track you for free. Advertisers pay. They don’t have to store data if no one pays. This of course in no way justifies the journalist losing his data.

The advertisers don't pay for google storing my data. The advertisers pay for ads displayed and ads clicked. How google does that is an implementation detail.

Google does that by storing my data, many kinds of my data. You making the distinction between data that "the user knows and understands" versus data that "is obscure to the user but advertisers care about" is a matter of marketing, in that as the former is easier to charge the user for.

My point is that the argument of "you're not paying for storing data X therefore fuck you" is a silly one, because you're also not paying for storing data Y, and neither are the advertisers.

> The advertisers don't pay for google storing my data. The advertisers pay for ads displayed and ads clicked.

That's like saying you don't pay the restaurant for the chef, you pay the restaurant for the food. It's a level of semantics that rises to being simply disingenuous. It's a restaurant, not a grocery.

No, it's not semantics, and it shows that you have no clue what you're talking about.

You can sell ads without targeting. In fact you can even sell ads without users.

Source: I worked years buying and selling online ads.

> You can sell ads without targeting.

You can, but that's not really what Google is selling anymore. It's certainly not the only thing Google is selling.

Unless you're making the claim that the data storage that Google uses for their data tracking isn't paid for by the targeted ad service? That if Google suddenly couldn't use that data for targeted advertising anymore that they'd still keep it around, or that none of their other services would raise in price?

>> You can sell ads without targeting.

> You can, but that's not really what Google is selling anymore.

Wrong again. Google's thing is search ads. Those target keywords, not user data. User data is a side-gig for Google, mostly thru youtube, maps, and gmail.

> but you're not paying Google to host your data.

My organisation (OpenStreetMap) is paying Google to host our emails.

They silently delete random ones, they never arrive at inbox.

(we are preparing migration to competing provider)

Paying does not help.

Which provider, if you don’t mind me asking?

mailbox.org (from reading ops issue)

> Paying does not help.

As I pointed out in the very next sentence of my previous post, yes, in this particular case the journalist was paying.

Yes, and I wanted to give another example that it does not help.

> * I personally ran a website with Google Ads, and the first time I tried to cash out was accused of click fraud and had all the earned money stolen,

I've heard of this happening to a lot of people. I hate using Adsense, and have mainly disabled it in the past, but it seems when I do, I get super heavy traffic and always think, "darn, I should have kept Adsense running".

I'm wondering if it's worth slowing down my sites and annoying users to eventually get to $100 to see if I'll be able to cash it out. It's getting close to $100, so it's hard to deactivate.

> but it seems when I do, I get super heavy traffic

Do you mean you get more traffic when Google Ads are active/present on your website?

I.e. that the google search ranking algorithm prioritizes websites running Google ads?

Maybe I misread but GP seems to be saying the opposite.

No, the opposite. When they're inactive, traffic reaches all time highs. When they're active, traffic is still decent, but nowhere near as high when they're off. Google ranks your site based on speed (as well as a few other factors), and their product (adsense) slows everything down, so that's potentially a reason.

So, it's like... wow, I should have kept adsense on (although I probably wouldn't have the same traffic if it was).

The Google Catch-22

If I recall correctly, if you choose to close your adsense account send you your remaining balance even if it is below $100.

As long as you don't turn on auto-ads, you can control Adsense to take up a limited amount of space as small as you'd like. Most of the users that would be offended by ads are already blocking.

Yeah, say all you want about AWS pricing, but damn if Amazon doesn't go out of their way to support things for a very long time.

What I can say about AWS pricing is that my monthly personal bill is just over $10 for all I want to do personally, and my employer's bill is... higher, but cheap compared to the cost of my time.

If I had to lose one bit of data or port one platform due to Google's !@#$%^&, it would more than pay my lifetime personal Google expenses many times over, as well as of the lifetime expenses of my current employer.

Former employer bubbled up to millions of dollars, and it was still worth it. Former employer got screwed over by Google to a greater tune, despite having a dedicated engineering support team at Google.

The math starts to work out differently for some of the PaaS providers, where costs are several times higher than AWS, and my time is cheaper despite the (rather impressive) time savings. However, having a reliable cloud provider pays for itself many times over.

I've lost had serious unexpected expenses literally every time I've been in any business setting which did business with Google. That's quite a few times. Literally the only time I'll do business with Google is:

- Free tier, quick one-off hack

- Legacy

- Literally no other options

Even the legacy, I'm working to port away since it's cheaper to do on my timeline than Google's.

Not only that but they actually provide real support.

I once complained on Twitter that I was having trouble with an EKS issue, and the product manager for EKS reached out to me directly, put me in contact with one of their engineers, and then they discovered an issue and rolled out a fix for it that day.

In a separate incident, I couldn't get GCP to read the verification data for a domain from their webmaster tools so I was unable to use the domain with GCP Cloud Run. I mentioned this on Twitter, and the product manager reached out to lecture me about how I shouldn't be using Cloud Run with domain mapping because that's in "preview". He then pointed me to another method to point a domain at cloud run, which it turned out was also in preview. All of this was because Google Webmaster Tools validated the domain but GCP wasn't able to get that information. I ended up cancelling the GCP project and just using AWS (as I should have from the start).

It's embedded deeply in Google's culture that they not only don't provide support, but they'll insult and gaslight the people asking for it.

It's not support if you have to complain on a public forum though.

That was due to my own whining and impatience. Going through the normal support channels tends to be pretty solid through.

I will say though that companies which take support seriously do monitor all sorts of channels as well.

> companies which take support seriously do monitor all sorts of channels as well.

Very much this. At my company we have many other means of support readily available, but some people instinctively reach for Twitter for peer support. Being active on Twitter means our support team can help even those people who don't think to use our official channels, which is good for the customer and good for our image.

Twitter probably isn't official but some companies have official social media support channels. Azure has an issues-only GitHub repo for Kubernetes (AKS) and you generally get a project team member instead of fighting through layers of support.

You don't need to, there are proper support channels.

I think that parent post was just implying that AWS cares about it's image and will provide support even if you don't go down the correct channels.

Google doesn't give a shit regardless of the method of request.

They do have support channels depending on your account type. I've used the enterprise support and they respond well.

They even go into your cloud architecture and tell you where you fucked up.

There was that time in 2013, that I leaked my AWS key and a "hacker" ran up a $3500 bill running Litecoin mining[0]. Not only was the support excellent and forgave the debt, Werner Vogel (AWS CTO) popped into the HN thread[1] and offered to help -- I got to email him and tell him his people were already on top of it.

And yes, they have made their money back off me in the 10 years since (average $100-200 a month).

[0]: https://vertis.io/2013/12/16/unauthorised-litecoin-mining/ [1]: https://news.ycombinator.com/item?id=6911908

Exact same thing happened to me but the bill was $11000.

Ouch, yeah it became really common, and I know of ex-colleagues that had the same problem.

Would you mind sharing how (in broad terms) your key got leaked? I'm always interested in stories like that as a point of comparison to my own (not great) security practices just so I know how reasonable I'm being.

It was a GitHub repo that I made public that I didn't realise had a key in it.

And yes,it's bad practice putting the keys in code to begin with, and I knew better.

Happens to the best of us Thanks for being willing to share, openness about "stupid" mistakes helps everyone else avoid them.

Biggest lesson is to setup billing alerts for any service like that. Put it 20% above normal usage and then you'll get a fairly immediate warning.

Won't save some of it if you're sleeping, but can definitely reduce the time someone has to run up bills.

>It's embedded deeply in Google's culture that they not only don't provide support, but they'll insult and gaslight the people asking for it.

Also to the users of their open source software, as many people who used Tensorflow 1.x can attest to.

> It's embedded deeply in Google's culture that they not only don't provide support, but they'll insult and gaslight the people asking for it.

This feels like the crux of the difference between Google and Microsoft/Amazon.

Google product culture seems to assume they built everything correctly, so if there's an issue, you're questioning their competency.

Microsoft/Amazon product culture seems to start with "weird things will constantly go wrong", so support is viewed as an opportunity to figure out how things can be made better, based on real world experience.

For most Google products, what seems to be missing is a link between customer support and SRE. I.e. "We're Google, we can't do customer support at our scale. SRE is a superior way of keeping services reliable."

Fair... but SRE doesn't handle "the real world is messy, and here's a situation a product manager never thought of."

Amazon Cloud Drive used to be “unlimited”. They rescinded that offer and deleted people’s data way before Google. Before Microsoft’s OneDrive too IIRC.

Out of the big three, Google Drive had an insanely good run as unlimited. (Even non-enterprise business plans with caps in theory were unlimited in practice until this year.)

- Former ACD customer who got burned.

They also banned encrypted files, effectively kicking off everyone using it for backup.

I assume this means they look at your photos and other data.

Probably it means they want to compress and dedupe and you cannot compress data after it has been encrypted.

Wait what? I'm a confessed ignoramus but this does not make sense to me. Encrypted data is bits and bytes like everything else. You can compress it. (Maybe not as well as unencrypted text, but still.)

>Encrypted data is bits and bytes like everything else. You can compress it.

You can "compress" anything, if one of the modes of your compressor is simply to store the data "as is", but generally speaking encrypted data is indistinguishable from a random bit stream and cannot be compressed.

If you think about it, one of the ways that compressors and attacks on encryption work is by looking for structure in the data. If you still have that after encryption, you don't have a good algorithm.

Encrypted data is highly random, and a compression ratio would be close to 1. So yeah- technically you can run a compression algorithm over it, but it won't help much

OK gotcha. Close to, but not exactly 1 because patterns appear even in randomness.

Because compression implementations generally have overhead, the output can actually be larger (similar to how compressing already-compressed data can produce a larger result.)

Yes, but the times when you found these patterns (and managed to get a smaller output) will be matched by the times you found no patterns (and therefore the output was larger), such that the average compression ratio over a large enough amount of inputs will tend to 1.

> Encrypted data is bits and bytes like everything else. You can compress it. (Maybe not as well as unencrypted text, but still.)

Due to the pigeonhole principle, to be able to make some inputs shorter you have to make some inputs longer. The way compression works is by making more likely inputs shorter, while making unlikely inputs longer. But for encrypted data, if your encryption method is good (that is, not ECB mode), all possible values are equally likely (unless you known the secret key).

Big part is deduping. If they have a million images that are identical they can just store one and symlink the others.

What? How can they tell the difference between an encrypted file vs. some binary format they simply don't understand?

There aren't really all that many common unknown binary formats, in terms of GBs in people's drives.

Mostly it's photos, pdfs, word documents, executables, videos, etc.

Aside from that there are mathematical tools you can use to detect whether a binary blob is compressed/encrypted or not, like entropy calculation.

Incompressible files that don't use a standard compression format (like a video codec) are usually encrypted

Let me share a Chinese Firewall (according to rumours) algorithm for detecting cryptography: count number of 0 and 1 bits. If the number is almost equal, it is likely encrypted content.

That also applies to any good compression though.

True, but since very, very few files are compressed with unique or unknown compression algorithms, you can read the metadata in the file and see if it's really compressed or if the whole thing is just seemingly random.

That gives me an idea. Might be fun to write a tool that will encrypt a file but make it look like a valid .zip file that just unpacks to meaningless noise. There's gotta be something like that already out there... Although you could recursively unpack/evaluate and eventually you should get to the point where it's non-random data. If you never do, then it's encrypted.

Makes sense.

There are some academic papers (I think one by Ross Anderson) which state that steganography relies on there being some randomness in the data you are hiding the message inside. So, to make the bandwidth available for for steganographic data hiding as large as possible, those needing to conceal data need incentives for innocents to publish files constructed using as large as possible quantities of random bits. It looks like Stable Diffusion is providing this incentive!

Sounds like a cat and mouse game - hasn't someone tweaked an encryption algorithm yet to produce a less even distribution?

They did? I have a bunch of encrypted files on gdrive

The person your replying to was referring to Amazon Cloud Drive

Amazon did? (ban encrypted files). I'm guessing so, just want to make sure.

I used to use Amazon stuff but it has really become such insane spyware. There is no data collection/analytics that goes too far for Amazon. It's really appalling, no less so that they use their near monopoly on using Kindles to deeply mine for data on exactly what people are reading, how long on each page, etc. Really gross.

To add to that, and related to,

>I've had customers get their cloud accounts shut down for a literal three cent billing error

I once just stopped paying an AWS bill on a small website I abandoned (few USD/month), came back to check after a long while and I was surprised to find out how long does it take for AWS to actually shut you down.

Also on AWS, on two different occasions I screwed up and got a high ticket bill:

* Got a lot of data out of Glacier and bill spiked to a couple thousand $$$ (this one'e explicitly mentioned several times in their docs but my mind just didn't register it).

* Forgot to turn off some expensive spot instances and left them running for ~2 days (where it should've been minutes). Again, some thousand $$$.

Both times I opened tickets, told them it was an honest mistake and they just wrote them off, :O.

During covid the company I worked at had their Series A fall through- we had a signed term sheet and everything, but the venture company lost their shit and pulled out of the deal. We only had a few months runway left at this point.

I reached out to our account rep at AWS and asked if we could delay our bills for three months. They laughed and said that if we didn't pay our bill for three months their system would just send us a bunch of emails, but that they wouldn't even start the process of closing the account until at least three months had passed- and even then they'd delay it further for us.

We got a new funding round and paid all of our bills, but we would not have been able to make payroll if AWS wasn't so lenient about it. That was 2020, so almost three years ago, and that company is still regularly featured at AWS talks and product announcements.

Same applies to Azure, they also have their issues, but not Google bad.

Just to be put the record straight: Google is an advertising company with hobbies.

And like for many of us hobbies change over time, so do Google's hobbies.

There's somewhere a website with all the products killed by Google for various reasons, some even after a few months.

@ExoticPearTree - "Google is an advertising company with hobbies."

This should be Google's mission, purpose, values and tagline.

The website you're referring to: https://killedbygoogle.com/

I've always wanted to ask, how do engineers who work at Google think about this? I want to work at a company where I feel like I'm making the lives of our users better. I'm sure this is true for lots of users of Google products, but there are also thousands or tens of thousands of people who have had serious real life impacts because of it. You can see many of them in this thread: AdSense businesses getting shut down, Youtube content creators getting banned for mysterious reasons. I try to work at companies that believe these problems to be unacceptable to even 1 person. While that goal may not always be possible, I would not be comfortable with anything less than an immediate resolution and a postmortem to prevent it from happening again. Is this also true at Google? I don't know anyone at Google to ask.

I think it's the same for everyone company: as long as the pay is good with perks and benefits you don't really care what you code on. And if you drink the company kool-aid, you believe you're doing people a favor.

Google decided it is time to sell GA for enterprises with 3rd parties involved. One client of mine (1B+ revenue, Xth largest website in Europe where x < 10) did not sort out the 3rd party agreement on time. Google's answer: we are going delete all of your data. Have a nice day!

this seems like there is something missing to the story though, no?

Yes. The sanity from Google.

It’s ~18 years since my blogpost blog caught click fraud, I still can’t do many things related to ads because of it. That credit card is still on their record and that address on which I lived total 2 months one and half decades back, is where I live in their record. Stupid stupid company.

Their abysmal user support is the number one complaint. How the f*ck didn't they wake up to this issue by now.

I think they are in dire need of a leadership change. If they want to serve businesses, they need to adopt actual business practices.

Support doesn’t scale so Google doesn’t want to do it out of greed and immaturity.

If any given user needs x support hours a week, and their business pays for > x support hours a week, then support scales.

I guess as you scale up a business you want to get leaner and you accept lower margins, hoping to make it back by volume? Maybe that's what you mean?

I assume he/she meant scale == grows (mostly independently) of human hours. Support can never scale; only fixing your software so that it doesn't trigger support needs scales.

Otherwise, each support request linearly requires human time.

> meant scale == grows (mostly independently) of human hours.

This is a pernicious usage of "scale" that has crept into tech-speak, because it makes the term strictly less useful.

Damn, I've had a great time using GCP at my last job, especially coming from AWS. This disappoints me a lot but I can't say I'm surprised, I've stopped using Google services almost everywhere (I'm slowly migrating away from Gmail, and I'm still stuck with YouTube) for this very reason.

I'm very aware that cloud vendor lock-in should be avoided (or mitigated) at all cost, but PubSub was actually pretty nice.

  I've had product managers for GCP lecture me that I shouldn't be using "preview" services (despite services being in preview for years), and then tell my I should have used a different service . . . that was also in preview.

If anyone’s ever been curious what working at Google is like, this sums it up perfectly…

External to Google, I think we have little data, but we've heard that much internal behavior is based on individuals making data-backed cases for positive contributions to the company.

Which prompts the idea: does Google have a data-backed adversarial operation, as a check on all that activity and biased analysis?

For example, when someone says "I did X, which resulted in a Y% positive impact to Z, so pls gimme dat sweet promo yo"... there's another function of the company that can say, "It was actually only 0.2 that positive impact, in that isolated regard; but more importantly, it had costs to A and B, which our current strategy model suggests resulted in substantial net harm to the company. Please tell all your colleagues that disregarding A and B is not the way to get a promotion."

It seems safer to not have any financial ties with google at all. Not paying for services and not being paid for services. Not publishing apps, not using ads.

-just a free user of gmail who hopes to be undisturbed.

I'd rather be a paid user of Fastmail than a free user of GMail.

>People really need to understand that Google is a garbage company that can't be relied on for anything.

I've heard horror stories with GCP from paying customers.

> company that can't be relied on for anything

Any cloud service can't be relied on for anything. There is no law for mandatory backing up cloud content and no guarantees your data will not be lost. Buy HDDs while they still are available and do the back up yourself.

Aren’t the guarantees the SLA that is agreed with Amazon, Google etc?

The journalist had nearly 500 TB of data to store.. getting that much local storage could be prohibitively expensive. That's sort of the whole selling point of cloud services.

Is it really better with other cloud providers? Genuine question. I personally never faced terrible issues like the one described, it’s always difficult to understand if it is really a GCP problem and not a general cloud problem.

Reply on news.ycombinator.com