GCP automatically lowered our quota, caused an incident, and refused to upgrade
twitter.com
twitter.com
Maybe that's how that salesperson or their org thinks?
Nah - it's just a numbers game. Sometimes they'll be wrong; that's a dead end.
Sometimes they'll be right. And some of those times, they'll make a sale.
Assuming yashg has long term memory and will talk to coworkers and other acquaintences in the future about this topic, the one case of being duped will lead to hundreds of potential customers being more skeptical of GCP in the future.
The one missed sale here undermines the future prospects of hundreds of possible sales.
One more trick used by channel partners it to move the billing under their name instead of billing directly with the platform. They promise to give a % of the commission they will receive from the platform back to customer. So they promise savings that way. They also promise to optimise your cloud infra, shut down unused instances, downsize some services that are not fully utilised and getting reserved instances for ones which are not yet on reserve. And for some companies it does end up with significant cost savings without actually switching the provider.
Now I have to know more! Did you call him on it? Did he have any sort of explanation/excuse?
Best network from all and coherent modern ui.
Not the usability hell like azure... (You know when clicking on a often used resource on the start page which let's you jump directly to it but doesn't allow you to jump a level up of all the other resources of the same type which totally works fine when you navigate to it the normal way... Or the huge hassle and complexity of resource groups for f everything...)
But you know the tweet not even states what quota was reduced.
My only problem with GCP is that their support is horrible, I much prefer AWS support that’s why I can’t use GCP beyond my hobby projects.
I’m trying out Cloudflare workers this weekend so we’ll see how that goes.
Their AKS offering was a crap show during the first year of general release and I opened countless tickets and they were snappy at the response times which is interesting seeing at the scale they operate.
Judging by all the comments GCP seems like one to avoid? Which is a shame because I had a desire to train multi cloud. If they treat their support like the rest of their products then I'll advocate a different IaaS where I'm able to.
I read in a much older thread that GCP’s TAM and enterprise support is pretty good:
> Hey, thanks for all that you’ve done. My experience with GCP has been an incredibly positive one. GCP documentation has always seemed fantastic. Our TAMs were very responsive.
> GCP support has by far been the best support experience. I have to say that the initial days it seemed to suck. The UI was some 90s google group clone which wasn’t even accessible through the GCP console, it was its own separate site which I always found amusing. But over time, the UI and quality of support became more streamlined and predictable, and I consider it one of the best SaaS support experiences today.
> One particular incident I’ll never forget is a support person arguing with me why network tags based firewalls are better overall for security than service accounts based firewalls. I expected to have a very cut and dry exchange but the support engineer actually did convince me that tags are superior to using service accounts. I did not ever expect to have had such a discussion over enterprise support tickets.
I’m pretty happy with azure. I’ve not used GCP but vs AWS I find permission management a lot nicer. I have yet to have a billing surprise that didn’t come down to me not reading closely enough.
What I don’t like is they sometimes lock private links (aka the ability to not have a service publically routable, only accessible on a vnet) behind premium SKUs, looking at you service bus.
I've been aware of some enterprise projects that are running on Azure, where from the description of how their architecture worked, I couldn't understand how they weren't drowning in network cost. It makes a bit more sense now if I know that Azure weren't charging people for some types of network bandwidth.
Don’t even get me started on a deployment story for GCP their deployment manager is deprecated and redirect you to use terraform.
I hate to swallow it but Azure was more usable and straightforward.
There is no world where Azure is more usable and straightforward. Just my two cents.
Not arguing the Azure UI is amazing, but it’s low on my list of concerns personally for cloud services.
AZ CLI is also terrible for interacting with Azure.
I have over 10 years of experience across all cloud providers and you just assume I'm a troll?
That's not a way to have a discussion in good faith...
I still don't mind upgrading a k8s cluster on the UI, monitoring it's healthy and than patching the tf code for the newest minor version.
Unfortunately for us, query queues only affect spikes in concurrent queries and not overall throughput. We’ve been struggling with bigquery support ever since.
So... we left and switched to Smarty and haven't looked back since. We spend tens of thousands on geocoding annually. Really mind-boggling behavior from GCP.
Currently my supposedly free micro instance on GCP is projected to costing 1.50€ per month. Obviously I'm very greatful for free computing, that's very nice. It's actually the main reason I'm even entertaining using the cloud for things and bothering to try things out. My complaint isn't about that, it's about the fact that their stated cost isn't the actual cost. It does seem to actually cost something. No, not network costs. It says "E2 instance core" and "E2 instance ram". It will be very interesting to see whether I get an explanation for this and whether they will take the money. In the first month they already took a few cents for the 2-3 days I ran it.
I don't know if irony is the right word but the GCP cost only has to go slightly higher and I can already switch off the cloud to a standard VM somewhere to have cheaper compute. Imagine that, I'm not a 100k/month spender like some here, I'm a free user and it's already almost cheaper for me to switch off the cloud.
Whatever heuristics cloud providers tend to use to discover and "remediate" such behavior is a totally different thing (e.g. obliterating an established account when an API key might have gotten leaked and a few GPUs let loose is a little overboard), but if I was offering a similar compute service, and its sole purpose wasn't just coin mining, I'd almost certainly ban it as well for similar reasons.
They can also use an absolutely incredible amount of resources, because it's not their money. Two hundred A100 GPU's for a week? Sure, why not?
Historically GCP/GAE has had a more generous free tier without credit card requirements, so they've always been a bit stricter on this than the other clouds.
Source: Worked at a cloud infra provider that struggled deeply with this problem.
Generally speaking you're not going to get banned unless you spin up max quotas and have other signals for fraud, such as zero payment history, connecting from regions with high fraud rates, etc. More often the provider will just limit your resources - either openly, like AWS does, or more subtly.
Unless that's exposed to the customer (AWS burstable instances are actually okay IMO), that sounds like the cloud provider committing fraud and hoping nobody calls their bluff.
> Any act of deception carried out for the purpose of unfair, undeserved and/or unlawful gain.
(https://en.wiktionary.org/wiki/fraud#Noun)
Is the cloud provider getting money off of this arrangement? => Yes, obviously they're getting paid here.
Is the cloud provider getting that money as a result of deception? => Yes, by selling ex. the use of 1 CPU core when they actually have no intention of letting you use that core. Now, I'll grant that in an actual legal case it might well be possible to get away with this by burying it in the ToS, but this is HN, not a court room, so I'm comfortable setting the bar at "would most actual users expect that to happen based on marketing?", conclude that no, most users would consider that surprising even if there's something buried in the ToS claiming that it's allowed, and call a spade a spade.
the worst part is that there's somebody in a GCP office somewhere that might secretly believe that I'm actually a cryptobro. shudder
you're probably safe on that front, it isn't like the GCP folks are paying any attention to the customers.
In either case (gym or GCP) you'll be entitled to a refund for the difference. They aren't allowed to just kick you out and pocket the difference.
They had billing accounts set up for all their projects. They were happy to hand over money but had no ability to.
https://twitter.com/JustJake/status/1667492928758095872
From the call with Google just now:
Them: "You exceeded the rate limit"
Us: "We did 5000/10min. The quota was approved at 18k/min"
Them: "That's not the rate limit"
Us: "What's the rate limit"
Them: "Not sure have to check with that team"
So, the quota is completely cosmetic...
Plenty of people care more about other job qualities than how much it pays, and a well run business can create a great environment without paying more than average. Look for companies where turnover is almost zero (assuming company is not growing) and you often find happy employees.
Places with worse employee conditions have to pay more, everything else held equal.
Anecdotally, I have certainly stayed in jobs with poorer pay because I liked my colleagues and the working conditions; but alternatively I have stayed in another job mostly because the pay was great.
I am an engineer working on such products and it was a routine on-call shift where I got this kind of question.
Admittedly we should have this cover by our support engineers, that are very very good, but this one slip through and I took the time to answer such query.
I am not a fanboy or anything, I could not be further from a fanboy or a blind fan.
But after working on AWS, I do suggest it as very sensible choice. Again, not because it is my employer but because they really take care of operations and customers.
Screw ups can happen but they are very rare in my experience.
Here is a ticket I opened recently (paraphrased):
> Me: I'm seeing an S3 charge in my billing and usage report called "AMZN-Out-Bytes" what does this mean? I already read the list of billing codes in this page (link) and it's not there.
> Support: Actually the majority of your report is "DataTransfer-Out-Bytes" charges which is caused by egress traffic to the internet. For more information see (link I already gave them).
> Me: Thanks but I don't need assistance with that billing code, please explain "AMZN-Out-Bytes." And can you explain how it's different from "AWS-Out-Bytes" as explained in (link)?
> Support: As explained in the page you linked, "AWS-Out-Bytes" refers to data transferred between different regions in the AWS network.
> Me: Please explain "AMZN-Out-Bytes."
> Support: Please close this ticket and reopen it with the S3 team. (who ended up answering my question)
In comparison, Google support could only be described as hostile, seemingly looking for any reason to close the ticket as my fault.
Neither, you're probably not seeking out such stories (there are plenty, even on HN). Google tends to get more negative attention on this particular forum, which might play a role, but the larger component is probably just chance.
AWS is _extremely_ customer friendly and if this happened would likely be offering dedicated support to make it right, credits for the business loss, etc.
Google's customer service is the worst I've experienced in the industry (like even speaking to a person is hard). While AWS is some of the best I've received.
Meanwhile we can't even get Google to answer an email about serious incidents that were very obviously their fault, much less assign a dedicated human point person (hell with AWS, we even had several backup contacts assigned to cover for vacation time). We were important enough to be featured on their client success frontpage, but that didn't make a difference in support quality we received.
I can't imagine why anyone would risk their business with GCE. Especially since it tends to cost more than AWS these days.
You're seeing culture in operation.
It doesn't take much legwork to find counterexamples of that claim, even on HN (see below, I spent 2 minutes searching to find those).
> Google's customer service is the worst I've experienced in the industry (like even speaking to a person is hard). While AWS is some of the best I've received.
This is a bit too hyperbolic for me, but on balance I agree that Amazon has better customer experience than Google. That doesn't really answer what GP is asking, especially given all the posts made about AWS on this very forum that only sometimes get attention.
https://news.ycombinator.com/item?id=2478129
https://news.ycombinator.com/item?id=25224220
If these are your examples of angry AWS posts then I think it just proves that AWS has amazing in customer service.
>https://news.ycombinator.com/item?id=2478129
That's 12 years old but a legitimate complaint about AWS customer service.
>https://news.ycombinator.com/item?id=25224220
Sounds like AWS had an outage, not sure how this is about customer service?
>https://news.ycombinator.com/item?id=35375558
That's a list of AWS customer's having security incidents and, again, nothing about AWS customer service.
>https://news.ycombinator.com/item?id=16283547
Not sure what this is about since the link is dead except due to being 5 years old but, based on the upvotes, it's someone failing to get people to have angry opinions about AWS? Not sure how this is about AWS customer service given the context.
> That's 12 years old but a legitimate complaint about AWS customer service.
Should have at least prompted you to do your own search. The fact that it didn't is the end of this discussion.
Though, when it might not be baked into the culture as much as at AWS, I don't know offhand how to reconcile a customer happiness turnaround with the rush to avoid being an also-ran in the AI deployment frenzy.
(Are you going to assign conflicting KPIs in a company that has cultivated a career-driven culture, and hope that individuals strike optimal balances for the company? Or partition the goal assignments to separate teams, when the optimal outcome requires holistic thinking and behavior across teams?)
Literally failing upward.
Amazon really is customer obsessed. If a department fails its customers that departments leadership will be replaced
I know they are trying to improve visibility on quota limits, and they have a tool now, but in my experience it was half baked and only knew about a handful of the limits we were running into.
Where it ends up being a bit of a nightmare is stuff like SMS on SNS where they would specify a quota but we could still hit it before their system reported it. We never did figure out if they were looking at a rolling monthly average and bursty campaigns could cause it to creep above their projection. This was always a manual review for this product and the only way we could avoid it was getting way higher quotas than we needed approved. Ultimately we ended up moving SMS over to Nexmo (now Vonage) for bulk SMS so we didn't have to have potential outages when the mystery quota of the now was reached.
They have a support page for quota changes and are usually actioned in around 15-30m on average.
What more do you want?
Only time I ever got pushback was trying to get them to make an unreasonable quota increase to work around a bug in early GKE that left stale network backend lying around, and when I explained they approved it.
I’ve found GCP support to be reasonable, but you have to pay for the enterprise tier. Their pricing model used to be insane ($250/seat??) but they fixed that a few years ago. I suspect a lot (not all!) of the complaints around here are for hobbyist / free tier, which sure, is garbage. If you spend $K/yr on support it’s fine though.
Gotta check out FBA/Amazin Marketplace to find all the nasty on Amazon.
AWS has to much hype riding behind it.
It’s not enterprise class and looks very incohesive.
Mine. And Gartner’s.
I use GCP and AWS, and AWS outclasses GCP handily.
Client: we were under the limit
GCP: No, you exceeded the limit
Client: Then what is the limit?
GCP: I don’t know
AWS were surprisingly helpful by comparison and even pro-active. Not a fan of theirs overall but much better than Google. This is a deep cultural problem with Google that also expresses itself in mobile development and everywhere: https://dev.to/codenameone/google-play-kafkaesque-experience...
They aren't the cheapest and their service is terrible. I can't think of a good reason to use GCP.
My team works with lots of companies, Google is one of the better ones in our experience. AWS is also excellent. I’m told Azure is good, but most of my experience with Microsoft is with enterprise product support, which is awful.
I haven’t used GCP support at work yet only as a hobbyist, sounds like maybe you had their enterprise support tier?
I have a hilarious situation at the moment where they decided to shut down my Workspace account because they retracted the grandfathered-in free tier. This has 15+years of history of a small business I ran in it. I tried to move it to a paid account, but because the account was created in a different country to my current billing country, the form doesn't work (can't enter address for credit card). And there's just literally NO way to do it. No avenue to any way to contact a human to quite literally give them money. So in the end I gave up : downloaded the emails and resigned myself to losing all the other history associated with the account.
But what's really hilarious is they can't seem to shut it down either, perhaps for the same reason. Every time they set a new billing deadline it warns me with another spate of emails and then the date goes by and its still there. I suspect their software doesn't know how to shut down a hybrid multi-country account either, and there are no humans so they are stuck.
The problem is that Google had no problem billing for a metric that they didn't expose to the user and didn't provide tools to debug properly.
2. It isn't up to google to tell you if you are querying against the cache or the DB. It's your code. You should know. Just tick something when you use the Redis/BQ/GCS/SQL client.
This task is so easy I would assign it to a junior engineer and expect code changes done in one day!
2. If the phone company charges me extra they tell me which numbers I called. Here a cache miss started happening. Only in production with no tooling available (at the time) to determine why this was happening. A single number of "data read" was all the information given. Not even the table name... That means you end up looking for a needle in a haystack.
I'm guessing you work for Google because your attitude seems similar. No they don't *have* to provide that service which is exactly why they suck. A service oriented company would make the *effort* to provide a user with this information. Especially a paying user at gold level.
Is that an option in AppEngine? The memcache docs seem to indicate the free tier has undocumented eviction policies.
It was mentioned that the service was artifact registery[1]. They were 5000/10min and approved at 18k/min [2] The ui appears to show a limit of 100 but not what of [3].
So reading the docs on artifact registery quota docs the only things that stands out is the "1 repository creation or deletion operation every 2 seconds."[4] so doing the math that is a rate limit of 30 per minute or 300 per 10 min so aroudn the ballpark of what they are. Looking at my own project this matches up and I can add request new quotas[4] but it needs to go through a human on googles end to approve.
Anyhoo if they were approved for 18k/min that is nearly 600x the normal quota. So they were likely (pure speculation) caught up in some kind of dragnet of quotas that were well beyond the norm. Skimming the site[6] it appears to be a rapid cloud based ide / deployment thing and I can't imagine how they are creating so many repos unless for each customer build they pass along to gcp's artifact registery 1-1. I use artifactory repo as well for managing docker images but mostly just make new tags off a handful of repos and am not typically creating them at such scale.
Sucks that they had the quota lowered without warning but "if" they were operating at the speculated quota that is flying pretty close to the sun.
[1] https://twitter.com/JustJake/status/1667660212902453248 [2] https://twitter.com/JustJake/status/1667492928758095872 [3] https://twitter.com/JustJake/status/1667478906591666176 [4] https://cloud.google.com/artifact-registry/quotas#project-qu... [5] https://console.cloud.google.com/apis/api/artifactregistry.g... [6] https://railway.app/
Google also tend to put algorithms first and people and their business second so that is why I would be very particular about using GCP for anything serious. We do use Google Workspace and for what we do with it, it is mostly ok, however rather expensive.
Blessed be the name of Google, come what may.
According to the general sentiment on HN, Google is being mean to developers and shouldn't get away with things like this, but Reddit merely has a rug and developers are silly to build things on top of that rug when it can be pulled away at any time.
I wonder why there's a difference here.
What makes you think they'll treat your small business any differently? You will be thrown under the bus the second you make them lift a finger to help you.
I think the users of this website just really like to repeat anything that fits the "Google bad" narrative, and if you don't have interactions with people outside of the bubble, it seems like Google services are on fire and practically unusable, but in reality it's fine for 99% of people.
- put price/quality pressure on aws
- fill weird niches made by regulatory capture and other nonsense
not using aws is like not using linux. there are valid reasons, but you don’t want any of them.
for most uses cases, netflix model seems like the right one. control plane on cloud, data plane on not cloud.
at a minimum you probably want to backup some high value data in s3 unless you have a more durable store somewhere.