Hacker News Reader

Has AWS support gone downhill in the last two years? I've worked with them in the past - as both an individual and a couple startups - I always reached a human. Issues weren't always resolved as quickly as I'd like but response times were short.

Working in small and medium businesses I've observed the same thing, and I've been quite satisfied with it. So I don't think it's really gone downhill, so GP's comment doesn't really resonate for me, but that isn't to negate their experience. Otoh I keep hearing horror stories about GCP and now I'm reluctant to try it.

Shitting on GCP is just popular on HN and always gets upvoted. AWS and Azure have royally fucked thousands of customers if you care to search for those writeups. My wild ass guess, considering posts like these have zero background details, is that they were careless with service account keys and their account got suspended for mining crypto or something. They also probably weren’t actually paying for support of any kind and that’s why no one is responding to them.

Can't say for GCP and Azure. But I run two small projects on AWS, with monthly billings of $8k and $150 respectively. I have always received very good support to any requests that I have made on both projects.

The last time I needed support from Google, I had found a bug in a service. The dude acknowledged it, got a request submitted to fix and provided me a with a coded workaround customized to my scenario.

Nope. We had been testing in our development and staging environments for months. We were deploying to production the exact same stack and we got our quota revoked within about an hour. We must have tripped some random thing. We have absolutely no idea what I could have been though.

Last time I checked you had to pay something like $400/mo extra for legit, timely human support from AWS.

I have the same question. I always got human response after 24-48hours or after one round of messages (with an automated human or machine, not sure). But so far, across 3 accounts and a dozens of correspondence, I always got a human.

I'm hearing that support experience has been rather enshittified by AI of late.

But that's only based on a bunch of anecdata.

It has!

In the 2010s I always got an AWS support team to help.

Now I get handed off to an external partner of AWS certified contractors.

They are often terrible. They have no backend systems access and just run through the AWS equivalent of "reboot it", "defrag your disk". Basically trying to find an issue in my pipeline. Which they never do because it's the same TF scripts used for years.

Only once we waste time going through the motions do I get passed up to someone who can actually correct the backend issue in the AWS stack itself.

Tbf though I rarely ever have to contact AWS support at this point. The few times I have in the last 2-3 was due to issues after they rolled out an update or with a newer service we wanted to use.

Never have issues with stable services like S3, ECS, EKS, or RDS.

My last experience was with someone giving me answers that where clearly in part LLM generated: It contained payloads/configs that did not even match the actual API

> They have no backend systems access and just run through the AWS equivalent of "reboot it", "defrag your disk"

To be fair I would bet money that the overwhelmingly vast majority of support tickets are exactly those kind of issues, and ones that refer to actual bugs on their end are, comparatively, extremely rare, and should have to be escalated through normal procedures to weed out common problems.

> they auto-reject your quota request if you don't respond to their emails within 48 hours but their responses take 1-3 weeks

It boggles my mind anyone would base their business on their good will. By now it should be obvious that companies with a huge number of customers don't care about individual cases that much for obvious reasons. That's why they cut on customer support. You get much better support with smaller companies where you (as an individual or business) are much more important to them.

if you want best support (while staying with big cloud) then Microsoft is the best .

Azure has its flaws but Microsoft puts a lot of people and effort behind it . We are not that large but there are so many instances where Microsoft reps will come in call with our customers or their people working with common customers will help out etc.

AWS has a done a decent job of taking enterprise business seriously last 10 years. you can get human support but generally they will charge you , I.e if better support you want you have to pay for premium support plans .

They are constrained unlike MS they don’t have non-cloud large enterprise business relationships for decades M365 or AD etc that helps with building the enterprise DNA.

In all three clouds it works best if you don’t buy directly, buy through a partner reseller , who both have the relationships to the CSP and have the people to work with you .

Sorry, no.

MS is the same network were even their lead engineers answer "well, uhh create a new account and hope you're not banned", when it comes to fixing a illegitimate ban issue.

None of the biggies are good. None of them.

You're better off building your own data enter. Can't believe I'm saying that, but I am. And it doesnt have to be acres and MW and water cooled. It can be a 42U rack.

Hell, I'm a homeowner and have 27U rack with 10U full, battery backup, solar, fiber and a backup internet connection, and stuff.

A small business could easy do this and own the hardware and software to their enterprise. In fact, they probably should. Helps prevent rug pulls!

Now I am curious what is the realistic price a business would expect to put down for a full rack. Say UPS, switch, 4-8U storage, and the rest CPU compute. Without entertaining GPUs, I bet you can get very respectably speced 1-2U servers for $5k a pop. So few hundred thousand probably gets you just an unbelievable amount of horsepower.

You can get a petabyte for like $14k now. Make a massive SAN. Even 10G networking is reasonable.

Compute? You can run stuff like Proxmox in commercial mode and get tons of features for really reasonable price.

Ram is now the big nasty, but I'm thinking with the 500 billion USD dropped on this circlejerk economy, ram will come back down.

You can also get a few graphics cards for AI stuff, but I'd constrain it to actual dedicated reasons, rather than some "AI everywhere".

Backups can be tricky. Run tape, but also run encrypted S3 backups remote, for 'holyshiteverythingsgone' reasons.

I used to be syseng for a small dev company. They had 3 racks by a local MSP, and was grossly mismanaged. Could have did everything in 1.5 racks. I have pictures, and you'd be aghast.

@dang why was my response autokilled?

Having run a small business on one of the big clouds for almost 10 years now building your own data center is insane advice.

>easy

Hell no

lol. I ran a relationship with a $200M EA. They suck. You need to pay for DSEs to juice stuff or drag in VPs to scream at if you are big enough.

The best way to get support from them is to buy it through a third party who can handle escalations for you.

As a petty person I would at least attempt to push this through small claims court.

Add up the amount you lost moving to another platform, value your time appropriately, and submit your claim.

A judge will decide if it’s fair, and you can be awarded ~10k in most places.

If they don’t pay for some reason you can put a lien on their property.

Quota for what? In my experience the GCP service quotas are pretty sensible and if you’re running up against them you’re either dealing with unusual levels of traffic or (more often) you’re just using that service incorrectly.

The quota we needed increased far beyond the usual was the YouTube API. The startup was a media editing and publishing tool, with a feature to upload videos to YouTube on your behalf. Uploading a video requires a ton of quota, which they gave us.

Regardless, dropping all quotas to 0 effectively killed our GCP account.

Interesting. I guess we’ve learned an important lesson in not building businesses around APIs that don’t have an SLA…

That was just a feature of the product to be helpful. Not a core function at all.

If you werent willing to pay for an SLA, and they clearly werent going to offer one to you… why is it surprising if literally no promises were made in writing?

Why would they intentionally lose money on your private commercial activity without even that?

> werent willing to pay for an SLA

According to who?

> no promises were made in writing

Most big businesses won't promise anything. That doesn't make their actions reasonable.

> Why would they intentionally lose money

You made this up.

Also if they were losing money on some feature, they could change the quota for just that feature.

Are you confused?

Clearly I was not asking for random role play on how another HN user may answer a direct question.

I'm calling out your "questions" as containing a bunch of unsupported claims about the situation on top of weird assumptions about how things have to work. It was not an answer, and your questions as written don't deserve answers.

How can your opinion even outweigh anyone else’s in the first place?

From what I can see there is no possible way your opinions could have some extra weight, above and beyond the median HN user.

(And in either case, you still seem confused for trying to initially hide behind some weird pretense reply)

> opinion

It's not an opinion-based claim. Maybe I missed something that would make me incorrect, but whether you made up details that make OP look bad is a factual matter. It's true or it isn't.

Also I said nothing about my opinion "outweighing" anyone else. Where did you get that from?

> hide behind some weird pretense

I'm sorry if it came off that way. I wasn't going for any weird pretense, and don't think most people would read the comment that way.

> It's not an opinion-based claim.

This sentence is literally an opinion.

Take your nonsense elsewhere, it’s totally derailling the thread.

> This sentence is literally an opinion.

I could disagree but it doesn't matter. I said a particular thing was not an opinion. You pointing at something else I said and calling it an opinion doesn't affect my claims at all.

What matters is my claim that you made up stuff about _drg9's situation. That claim is objectively true or false, not an opinion. And the evidence I see all says the claim is true.

> Take your nonsense elsewhere, it’s totally derailling the thread.

Derailing what? Nobody else has posted in this part of the comments in days.

This just seems like trolling now.

I cant think of anyone who cares if you classify your own opinions differently. 100% of your comments still are.

If you mean that more generally, that all comments are opinions, then you don't know what the word opinion means.

If you mean just me in this conversation, some of my sentences are opinions and some aren't.

How many services have meaningful SLAs for extreme downtime?

Github and (parts of) AWS will give you a small discount at 0.1% downtime, a bigger discount at 1% downtime, and AWS will refund the whole month for 5% downtime. But beyond that they don't care. If a particular customer gets no service at all then their entire $0 gets refunded and that's it.

> Quota for what?

Sure, I'm interested too.

> In my experience the GCP service quotas are pretty sensible and if you’re running up against them you’re either dealing with unusual levels of traffic or (more often) you’re just using that service incorrectly.

Well 0 is not sensible, and who cares if it's weird if they got detailed approval and they're paying for it.

Sure, but the comment is so vague I’m skeptical the OP knew what they were doing in the first place, or it happened exactly as they wrote. Maybe a service quota was reset to the default? But just set to zero? Doesn’t pass the sniff test.

"... and they're paying for it..." - that might be the exact issue. Google has no way to ensure that these small shops and startups will pay their bill, so quotas are used to prevent the company from running up a large bill they won't be able to pay.

I see a bunch of threads on reddit about startups accidentally going way over budget and then asking for credits back.

This doesn't at all mean the startups have bad intent, but things happen and Google doesn't want to deal with a huge collection issue.

If someone rolled up to your gas station and wanted to pump 10,000 gallons of gas but only pay you next month - would you allow it?

Well that is kind of a problem of their own making. The clouds refuse to entertain the prospect of pre-paying for services/having some sort of hard spending limits because they know that over-allocation is probably driving a decent amount of revenue.

I dont really understand ops problem as I've been able to set monthly limits on expenditure. Seems trivial to setup.

Are you sure you're not thinking of billing alerts? AFAIK there is still no way to set spending limits at GCP.

I'll admit it's a bit obscure But there is info here.

https://docs.cloud.google.com/apis/docs/capping-api-usage

Or you can do it programmatically.

https://docs.cloud.google.com/billing/docs/how-to/disable-bi...

That’s not how quotas work in GCP. Google sets quotas for certain APIs for interacting with GCP itself, like how many VMs you can create per second. They’re not billable. Sometimes these quotas can be be increased if you need them to be. But the way op described it makes no sense.

They denied my request for a service account quota increase even though my use case[1] was literally straight from their documentation. They only increased it after I complained on Twitter and got retweeted by Corey Quinn.

[1] https://www.agwa.name/blog/post/accessing_your_customers_goo...

Why didn’t you just have the customer create a service account and then send you the key? Or you’d just have one master service account and the customer would give you permission to impersonate the one they created? I’m sorry you ran into this but there were other solutions.

Having the customer send me the key is less secure because that key never gets rotated. Google wants to discourage long-lived credentials so badly that new organizations can't even create service account keys by default anymore.

Having the customer grant permission to a single master service account is vulnerable to confused deputy attacks.

In any case, why should I have to pursue "other solutions" to something that's in their documentation?

What did your account manager say about this. Getting this interaction right is the core of their job, enabling your business on the platform so you spend more money. With this bad an interaction I'd have asked for a new account manager.

A colleague had a similar quota issue. 4 times quota restoration request was rejected. Upon the final request he put “women owned startup helping underprivileged kids” and it was approved.

It can’t hurt.

Was the startup actually women owned and helping underprivileged kids?

have you met a vegan that doesn’t tell you they are vegan every day?

This is a stereotype, and I'd say a bit harmful. There are way more people complaining about such vegans.

I guess stereotypes are a conspiracy ? Besides, you get the illustration, obviously .

So it isn't I take it from your answer? If that's the case, not only did your colleague abuse protections for the vulnerable population, and possibly break the law by misrepresenting the legal aspects of your startup (IANAL), but you also didn't see anything wrong with potentially fucking over actual women owned businesses helping underprivileged kids (which I reckon is infinitely more valuable to society than whatever startup your colleague has) and even go as far as publicizing it on this high traffic website as a tip. This kind of behaviour is exactly the reason these protections exist, but big tech never gave enough of a fuck to do anything except virtue signal and democrats are too impotent to actually make them do it.

Call it prompt engineering

No I will call it what it is — fraud.