Comment by JPKab | Hacker News Reader

Comment by JPKab | Hacker News Reader

This is a great article, and sums up my team's frustrations with AWS Lambda and the AWS engineering leadership in general.

I worked in the past under a slew of former AWS product managers, and I know from them how the AWS roadmap works:

1.) Real, crucial customer feedback is incorporated into roadmap, usually for the core services that are already used extensively like Lambda. 2.) Executives have new, novel, sexy flavor of the month projects like "we need to catch up with that ChatGPT thing my kid was showing the Au Pair the other day!" 3.) Product Management leaders are incentivized to prioritize said new/novel projects, and proceed to do a worker crushing sprint to build what any sane person would call a PoC. 4.) PoC is barely tested and pushed out as a half-baked garbage product. Garbage product languishes. (See AWS Data Science Studio for a great example of giant, useless POS)

Meanwhile, developers are siphoned from real projects like, gee, I don't know, getting Python 3.10 out on Lambda? Or how about when you launch a sagemaker notebook instance you still top out at Python 3.8???? And don't get me started on recent versions of CUDA being available on GPU instances.

My team is relegated to docker containers deployed to lambda due to the antiquated, ridiculous limits on layer sizes for any of our ML oriented lambdas.

I have a buddy who works at AWS and talks about how utterly inefficient they are due to horrific leadership. At this point, you have a choice between a few highly dysfunctional cloud vendors. Google just SUCKS, Azure is a blatant financial rip off, and AWS is always developing new features nobody wants while ignoring the ones we do.

From my experiences working there, AWS is a feature mill. They've built their entire sense of self around shipping features. Stuff for Jeff Barr to blog about (that's no diss on Jeff, I think he does an awesome job). They can't get off that horse they've decided to ride because they're concerned if they do they'll be seen as having stopped innovating, but they're also not willing to provide headcount to meet both innovation goals and properly look after what they've already got.

> that's no diss on Jeff, I think he does an awesome job

Honest question: How is Lambda runtime version rot acceptable to an "awesome" top exec like Jeff Barr? Is someone else demanding he siphon resources away from maintaining key services? What data-driven metric says maintaining service quality shouldn't be continually prioritized?

I'm open to any explanation, but the logical conclusion I see if the lack of prioritization only really makes sense if internally, Lambda is considered a gimmicky toy. Where is the "Customer Obsession" mantra here?

A large portion of the most valuable work in software is of the boring, unsexy variety needed to keep key pillars working smoothly.

They’ve provided custom runtimes which removes the need completely for managed runtimes giving you whatever version you like. They also did it in a way with Docker that’s simple and powerful.

If I was an AWS exec I’d say managed runtimes are legacy, with the introduction of containers move to containers for new deploys.

I’ve been critical of Lambda in the past and critical when people use Kubernetes/Docker because it’s Kubernetes/Docker. The container support in Lambda has been a game changer, it’s one of them now they’ve done it you think why didn’t they think of that earlier, it’s a great fit.

> If I was an AWS exec I’d say managed runtimes are legacy, with the introduction of containers move to containers for new deploys.

If they do that they remove one of the key benefits of AWS Lambda and that's that customers don't have to care about the underlying operating system and just have to care about their function code.

And I'm sure if they'd would look at the numbers of how many AWS Lambda Functions use zip packages vs. Docker images they'd see that something like >90% use zip packages. Based on that they'd be insane to deprecate the use of zip packages.

Of course 90% will use zip packages, container support is relatively new in the lifetime of Lambda.

Deprecating zip packages doesn’t mean zip packages no longer work, it means pretty much what they’ve done, ignored the managed runtimes with the view if the current runtime doesn’t do what you want, make your own with exactly what you need using tools you are likely already using with no vendor lock-in.

For things like Python most people don’t care about the underlying operating system from the projects in multiple orgs I’ve worked in. They go to Dockerhub, type Python, pick the image with the highest downloads and run pip install inside it. If you use Go, Rust, anything that creates a static executable you use “FROM scratch” with the only thing inside the container being the executable.

Those who do care about the underlying image are likely large orgs with a set of already blessed base images or people using things like Nix to build containers.

From a large org perspective and developer experience getting a “registry” for free because packaging as a standards compliant OCI image means instead of effectively creating your own registry on top of s3, you get a far more robust one with tagging, asset hashes, layering etc out the box. I never had a issue with zip archives but containers removes a bunch of roll your own boilerplate registry.

This sounds like it would fall under

"Earns trust"

For the Amazon leadership principles.

(From an Amazon cdo engineer looking over at aws, nothing official of course)

Developers gain trust in AWS lambda by having it support the current runtimes, and keeping the old ones running for a long time. Lambda will always run the functions you want it to.

For some tension:

When I've taken free beers to chat with the lambda folks, it seems like they have a linearly growing oe burden for each new runtime version they support, and at least 4 or 5 years ago, I couldn't see a path towards avoiding that

Before my current role, I had to juggle roadmaps and integration partnerships with the Big Clouds. It's insane how much they ship their org chart but also their internal politics?

I am convinced that some of the teams there are really trying to angle for releases that ship for PR rather than the feedback. (Although some AWS PMs really are trying despite everything else.)

> Meanwhile, developers are siphoned from real projects like, gee, I don't know, getting Python 3.10 out on Lambda?

Why isn't competition putting pressure on AWS? A cursory googling around seems to show that Google Cloud Functions is at Python 3.11 and Azure is at 3.8. Which then makes me wonder why people don't use Google Cloud more? And why would anyone use Azure at all?

[0] https://cloud.google.com/functions/docs/create-deploy-gcloud... [1] https://learn.microsoft.com/en-us/azure/azure-functions/crea...

> Google just SUCKS, Azure is a blatant financial rip off

If AWS is the least worst option, and also the most popular, what motivation do they have to improve?

I’m a very happy GCP user. Cloud Functions are ass but Cloud Run is amazing, and as far as Python versions go, if you can make a Docker image of it you can use Cloud Run. Scale to zero. Automatic load balancing. Any Docker image. Yes please.

I mean has AWS ever been easy to use outside of S3? If they were simple, companies like Vercel and Heroku would not even exist (both runs on AWS infrastructure).

I feel like SQS, API Gateway, DynamoDB, (largely hidden) SimpleDB, and vanilla Lambda and EC2 are pretty damn easy. (IAM is admittedly semi-baroque layer holding them all together. I struggle to call that easy.)

The complexity and amount of AWS-specific knowledge required to efficiently use it is pretty much absurd.

Even though AWS offers multiple different solutions (AWS SAM, Elastic Beanstalk, AWS CDK, AWS App Runner) that in theory should make consuming AWS services easier, these options still assume you understand and know how to use AWS.

In my previous job, I've been tasked to create an abstraction that will make using these AWS services easier and more developer-friendly.

I failed.

Yet I realized how hard it is, and how many people/companies are struggling with it.

That's why I've founded https://stacktape.com/. It makes using AWS simple to a point where any developer can efficiently use it. Let me know what you think.

What are your issues with Google cloud?

I had a horrific support experience due to a product (it was a predecessor to Colab that they eventually just cancelled) that was (due to a bug that they eventually fixed) triggering the startup of expensive instances randomly that cost hundreds of dollars a week to run. I couldn't get any help, until I eventually out of desperation found a BigQuery evangelist on a Reddit forum who escalated my issue.

Venting rant below:

It's blatantly obvious that Google's culture is completely broken, and they are incapable of hiring people who don't think they're too brilliant and amazing to be customer support. They care about their advertising printing press and everything else is secondary.

Tensorflow was the first out of the gate, quickly got its ass handed to it by PyTorch, and then they just imitated Pytorch, but in a crappy, crowded API kind of way. Angular is hilarious in a sad, pathetic, "wow that guy still thinks he's doing something useful, doesn't he realize nobody cares" kind of way. Dart..... well, I don't even need to say anything more. I bet there are people on HN who deeply regret investing time in learning it. Colab: "We have paid plans! If you pay, it will be only semi-useless!" (I tried the paid version, utterly useless compared to Paperspace)

If anyone from Google is reading this, sorry, but you work at a company that is rotting like early 2000s Microsoft. Even my highly non-technical wife was asking me the other day if I had noticed how shitty the search engine is now.

BTW, am I the only one who bought a Google Home speaker and used to listen to Youtube podcasts on it, only to have a software update make it so that now I can't even use Chromecast to send any Youtube video's audio to my speaker? Really awesome Google. First time in my life I've purchased a product and had it literally lose features a year after I bought it. Also, Google Meet sucks, and has become increasingly bug ridden over the years.

> BTW, am I the only one who bought a Google Home speaker [...] and had it literally lose features a year after I bought it

There are tons of other features that are rotting on Google home/assistant. It used to be that one could set a location based reminder on the assistant ("hey Google remind me when I get home to do the laundry") but they removed that, for example.

Whooowee, pull up a chair and grab a drink

Their IAM is a raging tire fire. Disturbingly stupid for such an advanced organization. I'm on my phone so the rant will have to be left at that for now, but people can bitch about AWS IAM all they like, at least it was designed and thought out versus the GCP "works on my machine, ohh I got promoted now it's someone else's problem". Words fail me how angry GCP IAM makes me

I know this isn't a "cool kid" thing to say but the lack of CloudFormation makes deploying into other people's GCP more painful than necessary, which has an intersection with my IAM comment because Role bindings are authoritative (in both gcloud and terraform) so oh, you want to add a member to `role/editor` but leave the rest of the members alone? Tough! I had high hopes for Deployment Manager but it doesn't support hardly any of the resources and there's no one click URL to roll out the yaml anyway

And congratulations, you just created your first Project in GCP: now you want to run Cloud Functions? Well prepare to spend the next hours of your life turning on the necessary APIs for that project because unlike AWS where everything works to your credit limit, GCP is that blank room in the Matrix. For ever project, every time. It's like GCP has amnesia. What APIs are necessary, you may ask? Who knows, just try something, read the 403 it gives you back, rinse and repeat

Reply on news.ycombinator.com