AWS staff spending ‘much of their time ’optimizing customers' clouds'

Agreed that their general support (and some of the design and implementation of their services) makes me want to give up and go outside and touch grass instead of working in tech, but as soon as you reach the level where you are in contact with support directly, they're very responsive. It's a weird dichotomy.

Customers are able to report literally any issue at all via support. AWS support is a huge organization. On my service team, there were plenty of instances where bugs were reported by customers.

It sounds like you didn't use AWS support but you got AWS certified consultants.

You gotta love these assertions at the top of a post that flag people who confidently are saying they have no idea what they're talking about.

You can report bugs via your solution architect, via support, or via the Re:Post bug report tool.

> AWS support is atrocious, unless you cozy up to them somehow to get special treatment. If you are a "regular" customer, you get zilch. AWS offers best performance for the best price, that and they are the largest service provider is what makes them succeed. They don't need to spend anything on support -- it's not cost effective.

Even basic support is great, I've had most of my issues fixed within 8 hours. If you pay for business or enterprise support you'll have a dedicated person you deal with. Try comparing that to GCP...

> My personal experience: we provided a service through AWS marketplace related to storage. So, we are not your "regular" customer, had to pay them some for accreditation... and here's the thing: it's probably something about how we create EBS snapshots, but about every thousand or so snapshots creation stalls. Doesn't fail, doesn't succeed, just hangs forever. There's nobody to talk to in AWS about this. People who "certified" us (what a joke...) cannot do anything about this problem.

If you were a solution partner you would have had a partner solution architect who would take point on this problem for you. This sounds like your organisation didn't accurately share access to this resource with the right people.

> And there's more stuff like that. Sometimes some resources won't delete first 1-2-3 times you try. Some other resources are created, but don't work once in a blue moon. And all you can do about it is just curse and try again.

I've been AWS facing for 11 years and I can count on one hand the times this has happened. Did you file a support ticket?

Source: Work in an AWS consultancy, before that was running tech for a startup on AWS

This is not true; we regularly open tickets with AWS for enhancements or bugs.

Not only that, but we were proactively contacted by an AWS account team before we had even spent our first $100, and they spent many days providing guidance and support to our team before we even breached $1,000/month.

It may not be the norm, but my experience with Azure has been exactly the opposite. After having an (admittedly small) Azure presence for 4 years now, I'm going through the "front door" of their sales site to try to get an account person from them. So far, I recommend AWS over anyone else 100 times out of 100, 10 days a week.

Imagine getting that kind of support from any of the competition. GCP support was laughably inadequate (apparently they have separate support for Firebase and GCP, and GCP can't support for Firebase), while Azure support is exactly as you describe - great for enterprise, sucks for basic support. As for the innumerable AWS "startup" competitors like Digital Ocean , Fly.io or Render, support ranges from condescending to non-existent. In one of the above cases, support tickets were so backlogged, I was giving support hints to guys stuck on issues for days.

For something like our HFT firm, we see high sustained utilization so we have an on-prem cluster of nodes that we run. They're faster (more powerful CPUs), have more RAM, and you can run fiber optic interconnect that's, in practice, faster than on AWS. We still use AWS extensively, though.

So, for example, things like PXE boot, which would be useful in your own datacenter, but don't work (well) in clouds s.a. AWS become mystery. Obviously, this isn't limited to just PXE.

And since public cloud's goal is to provide service to as many customers as possible while having as few staff infra engineers, the total expertise among general public disappears. Having worked in the infra business for a while -- it's increasingly more difficult to hire infra engineers.

There was a point not too long ago when I managed a 6 engineer team responsible for 7 figures of monthly spend. Still seems outlandish to me despite the ops burden being pretty low once we had things sorted.

AWS spend = 500k * 12 = 6M

AWS spend / Total spend = 6%

Where the hell does 0.1% come from?

of course often folks just ignore costs until al of the sudden it's a huge pain point.

I write event (think food festival) software. 9-10 months out of the year there is almost zero traffic, 1-2 months have mild traffic, ~2 weeks are higher traffic, and then 1-5 days (during the event) are very busy.

I'd be hard pressed to find anything other than my current stack (Lambda/DynamoDB/S3/Route53/APIGateway) that costs as little. In the off-months my costs are ~$5 if that and in the month of the event that might get as high as $15-30. I cannot imagine being able to host it locally/colocated (for the month of the event) for that cheap and that doesn't even touch on the hardware cost (I'm thinking just electricity and co-locating costs).

Yes, I'm a small fish and this is just my side business but for me the cloud is way cheaper. Anything with very spiky/uncertain load can be good candidate for the "cloud" but a big part (IMHO) of making the cloud work is using the managed services and using them smartly. If all you do is spin up EC2 instances then no, the cloud is probably not going to be cheaper in the long run.

However, unless you use HUGE amounts of compute, cloud is probably the right choice, because you probably save a tonne on salaries by going the cloud route. It's way easier to properly operate a managed DB than your own (especially in terms of no-downtime upgrades, no-downtime scaling, backup/recovery, etc.), same goes for managed file storage (S3 type things), same goes for managed pub/sub, same goes for managed K8s/whatever, same goes for managed load balancers, etc. For a lot of startups, total cloud spend is equivalent to just a few fulltime Engineer salaries, and they also use a tonne of cloud services, getting that all running well without AWS (or Google Cloud, Azure, whatever) would take way more than a few fulltime Engineer salaries.

FWIW, the last company I worked at (~1000 people), as our AWS bill was getting into the millions annually, we tried migrating off AWS, onto a colocation setup. I didn't work on that project myself, but my understanding is they spent a bunch of engineering effort on it, were still nowhere remotely close to being able to truly move off AWS, ultimately canned the project and moved all the colo stuff back to AWS. I think a lot of engineers underestimate how much effort it is to provide the kind of cloud services Amazon/Google/Microsoft provide, at comparable levels of reliability and ease of use. It's A LOT of effort.

A) a user base with usage that varies significantly over time

and

B) an architecture that can actually scale significantly up and down during the same time period

So if you have lots of web back ends and business logic and general workers for jobs that can just be shut down when there's no capacity needed, there's a ton of savings available in the cloud.

If you have a big monolithic architecture or some other organization where everything is just on all the time, the cloud makes less sense as a cost savings.

I once wrote an internal chat bot which was used less than 2 seconds per day, but was actually supremely useful as it really simplified some process or another and took a fraction of a second to run each time. The actual cost to run this in the cloud was approximately 1 cent per year. If I didn't have this cloud capability it would be hundreds of dollars per year to host or thousands of capital in machines and maintenance to host myself.

The systems are scaled up for the "Black friday" sales event.

With cloud, you will plan for the steady state and just boost it up when you need to.

The anti-pattern is usually that when on-prem hits 80% load, some engineer is usually tapped to weed out all the slow code to push past procurement delays.

While in cloud, people just throw more hardware at it and ignore low hanging performance problems which are costing them money (& unlike the on-prem, fixing it will immediately reflect in the budget the next day).

Not sure if you consider small startups to be realistic scenarios, though, so perhaps this doesn't count.

Their peak demand is insane, especially at race start, and is only sustained for a few hours for ~20 weekends a year.

AWS have already built a live streaming stack for their other customers with DRM and support for a wide variety of platforms. And there are other features like live rewind, restart, live to VOD etc.

It is also relatively straightforward to serve new countries, you don't have to build data centres all over the world, just stand up your infra in a new region.

Bold assertion. Care to shed more light? Cheaper in terms of your labor costs?

But EC2? Configurations are static, cost optimisation is hard, the spot market is a pain, reserved instances aren't fun to trade-off. Compared to GCP where you get sustained usage discounts which are much simpler to reason about, and you can spec machines in much more dynamic ways.

And Lambda? It's just not a great environment to build applications for. Frameworks are necessary to tame complexity, API gateway adds a ton more complexity, dealing with any state is difficult. Lambda doesn't seem any better than any of the other FaaS offerings. Fargate seems like a better option for most people who want to use serverless, but again that's not unique.

The attack only lasted a couple minutes and I was able to weather the storm just fine. I only used it as an IRC bouncer. During the attack, my IRC ping went up, but nothing crashed until I got disconnected, couldn't reconnect, and had an e-mail in my box from DO saying they took my droplet down to protect their network.

you'd be shocked to know how many people believe that this exactly what the "cloud" is. these are the people making the support requests.

Show me your “agnostic” solution and I can show you plenty of ways that you tied yourself to your choices. Not to mention all of the organizational constraints and coordination - architecture review boards, planning committees, project management organizations, etc.

For instance, I want to do multi region deploys for lambda, I can't just do a simultaneous push to deploy across regions. I'm expected to use this messy switch[0] between the CLI and the CDK to do anything useful. This is a tiny example.

Not to mention the CDK packages have no comments in the shipped files (at least for typescript) so I can't even introspect the typings and try to understand the purpose of things. The documentation is atrocious.

Consider instead Firebase, Supabase or Fly.io, which are dead simple to use.

Developer Experience needs a major overhaul. It should be as easy or easier as Supabase or Firebase. No, Amplify is not a solution, its too buggy and has zero margin for customization plus, its local dev experience is quite bad

It'll probably never happen though, because developers are just end users, not the true customer, the buyer (e.g. management) is. I think that's the real reason why these improvements languish

[0]: https://aws.amazon.com/blogs/compute/deploying-aws-lambda-la...

You save a lot of time and resources just using SQS instead of setting up your own servers. And the lock-in is not as problematic as Lambdas etc.

It doesn't require much code change to move to an alternative message queue (RabbitMQ etc.), so the lock-in is minimal. In my projects, I just need to change a few lines of configuration code and the actual message processing code should remain the same.

Especially if you need a couple of racks things get complicated quickly. If you have never racked machines for a week you probably underestimate the amount of time it takes before you can actually run workloads on them.

And if you need a 1000 machines for x years which otherwise you'd put in your own data center you can absolutely go to AWS, GCP, OCI and bid them down against each other.

Big enterprisey applications like this would love to only be doing 10 queries, 30+ is much more likely, and I bet they won't be moving from 1 DC to 1 cloud region, so 10ms is also unlikely. Back of the envelope suggests more like 1s of additional latency.

That would barely work for a big enough OLAP deployment, with all sorts of varied consumers, let alone OLTP.

Counterpoint, the ops required for your own infrastructure has greatly dumbed down.

It might surprise you, but OEM hardware solution have greatly evolved and simplified the same.

Now I can setup a core switch running 100 GBs with failovers by just plugging in some cables between them, setting a flag and they immediately start replicating config between themselves.

>Even for good ops people, it can be very hard to capacity-plan and understand what appropriate/affordable/useful hardware is. Most ops folks today have probably never purchased servers for production deployments.

I'll shock you once again, the same way AWS has sales and support departments that help spoonfeed what you need.

So do the hardware OEMs. I can get complete solution walkthroughs with Dell for example.

You still won't be able to do it with idiot monkeys. You just have to enable a small team of strong people. It may be getting harder to find those people, but it's not so bad if you only need a handful of them.

https://oxide.computer/ is shooting for this space, from what I can tell?