Dear AWS, please let me be a cloud engineer again
lucvandonkersgoed.com
lucvandonkersgoed.com
This tagline is representative of every part of the hype around GenAI. It makes it sound like security has fundamentally changed and we all need to re-learn what we know. Everything to do with GenAI is treated like this: we need new security plans, we need AI Engineers as a new job title, we need to completely reevaluate our corporate strategies.
Security in the world of generative AI is not substantially different than infosec has been for a while now: User prompts are untrusted input. Model outputs are untrusted input. Treat untrusted input appropriately, and you'll be fine.
The same goes for "AI engineers", who are in the business of wiring up APIs to each other like any other backend engineer. We take data from one black box and transfer it to another black box. Sometimes a black box takes a very long time to respond. It's what we've always done with many different kinds of black boxes, and the engineering challenges are mostly solved problems. The only thing that's really new is that the API of these new black boxes is a prompt instead of a deterministic interface.
Don't get me wrong, there will be things that will be different in the post-LLM world. But my goodness do the current crop of companies overestimate how large that difference will be.
If you go to an AWS event in the future, the name of the chalk talk was "The Life of an IAM Policy"
I think the problem is they're trying to introduce nuance and a narrow path to allow this. They want an acceptable level of risk to using untrusted model output for the efficiency/productivity gains it will bring, notwithstanding hallucinations.
Generative AI would not have flown in the security theater of Yesteryear, but CTOs see productivity multipliers.
The CTOs are hallucinating as much as the LLMs are.
Another big area of hype is "prompt engineering." That one seems to have calmed down slightly, but for a while, there were large swaths of the Internet who were amazed that the set intersection of "talk like a decent human being" and "be precise in your communication" could generally lead to good results.
In many ways, "AI" right now is magic marketing sprinkles that you can put on anything to make it more delicious. (Or, if you're inside a big company, it's magic prioritization sprinkles.)
"Google-fu" sounds like a fun skill to learn and acquire, where "prompt engineering" sounds either like something well out of reach or like pretentious nonsense depending on the audience.
More likely, "prompt engineering" is a marketing term made up by AI marketroids (cf. androids) hoping to make developers feel better about their reduced roles in this "grand new AI age".
I don't think so.
I mean, clearly calling it "engineering" threw some people off, in the same blend as some gatekeepers cringe at calling train drivers "railroad engineers". But that's puerile gatekeeping that misses the whole reason why there is a vast need to know how to "engineer prompts".
The truth of the matter is that the focus of "prompt engineering" is being able to put together inputs that solve business problems in professional settings. You need to have full control over the generative process to integrate it's output in a business setting. That requires specialized knowledge way beyond naive requests expressed in natural language.
Complaining about "prompt engineering" because that only focuses on specifying queries and operating a specific service makes as much sense as complaining about SQL/database/postgres engineering because that only focuses on specifying queries and operating a specific service.
Before trying to dismiss "prompt engineering" through gatekeeping logic, first you need to justify why there is no need to know what you're doing to get outputs by feeding the right inputs. Even in subreddits dedicated to using generative AI to create images and videos,they started to outright ban posts where the contents are posted without the prompts used to create it.
Cosign prompt engineering. My startup is tl;dr "what if i made a on-every-platform app that can sync and let you choose whatever ai provider, and you pay at cost. and then give you a simple UI for piecing together steps like dialogue / ai chat / search / retrieve / use files"
Seems to me the bigs are completely off the mark, lets cede the idea there's an omniscient AI available. Literally right now.
Cool.
It still has no idea how you work.
you could see 42, in hitchhiker's guide the galaxy, as a deep parody of this category error
That’s just not an interesting or rewarding way to interact with a computer, and the last thing I want to do is add long wait times and nickel-and-dime cost to the process. Layer on using different LLMs for different tasks or trying them out against each other and cross-checking output and it’s a mind-numbingly indirect way to get anything accomplished that in the end teaches me nothing and develops no useful skill that I enjoy practicing.
If it works for you, great, but even the most honest and genuine fans make it sound like a nightmare to me.
It doesn't seem likely an LLM will ever do that. Maybe at a certain point of sophistication? But if the model is regularly changing - which they almost all will be, if they're expected to be up-to-date - there is a strong change they'll be different every time they're used.
(I've been getting different behaviour in even relatively narrow ML-based systems for years. Google Assistant is my prime example - I regularly use the phrase "add to my calendar on the 20th of September at 5pm, go to the park". Almost all the time, it works perfectly. But a couple times a year at least, it won't process this as an action - it just does a Google web search for this string.)
So yeah, prompt "engineering" is indeed a silly term, but software "engineering" kicked off the dilution of that word ages ago. And GPT models can be inspected and measured for input and output, prompts can be analyzed for their effects and usefulness, temperature settings even directly control some degree of determinism. It's not like models change on a whim unless you're just using end user products. Anthropic, Huggingface, AWS, OpenAI, they let you pick a release model version in your API calls and stick with it for a long time. If you're self hosting a fine tuned Llama 70b, nobody will ever force you to update it if you get it doing a task to your expectations. The quality of deterministic behavior in AI is currently lower than that of Excel or C code, but it's also serving a wholly different purpose, people want it to be creative and create novel nondeterministic outputs, comparing them is a bit silly.
I think of it as similar to Googling in the early days. What started as a skill I had to pick up became second nature and I could find things faster than my family without even really thinking about what I was doing. It just became natural.
Most of my colleagues communicate with chatgpt in broken english, or they ask a question while leaving out crucial details about their problem. They’re always surprised when i am able to get a useful response from chatgpt when they couldn’t. it’s comical sometimes.
I 100% hear you on the “not a fun way to interact” though. To each their own. I personally enjoy it, it’s like a rubber duck that can actually talk back. :) not for everyone though.
The problem is that GenAI is a complete black box with nondeterministic outputs. I can write code and I know with a very high degree of confidence what I expect it to do. Asking an LLM or a generative image program for something, I have no idea what it'll give me. It gives no feedback other than results, which may or may not be what I want. If not, I have to reverse engineer what I think it might want me to say in order to get desired results. And the same query placed another time might give a completely different answer. I don't deny that it can do some impressive things given the correct inputs, but I am not inclined to spend my time searching for the magic words.
You're showing a fundamental misunderstanding (or ignorance) of the whole problem domain.
For starters, you place an awful lot of emphasis on what you think is "carefully craft English language prompts". That makes as much sense as characterizing the job of a database engineer as "carefully crafting quasi-English language prompts". The language used is completely irrelevant, and being able to use in some circumstances something resembling natural language to build up context does not take away from it.
Any remotely honest and objective analysis of the topic would start from similar activities, and to start off the areas of work where Llama are being used. For image/video generation you need to look at graphics design, video editing, video production, illustrators, etc. These activities, by their own nature, are iterative and exploratory. Then for text you have the work of copywriters and editors, and even writers and essayisgs. The work is fundamentally iterative and exploratory. Then you have work like exploratory data analysis/statistics/data mining. Every aspect of that work is iterative, even the reporting part.
So yes, the actual question for software engineering would be how to get AI to produce and iterate on an OS. The hallucinations aren't the only problem then, the lack of predictability in the answers is the biggest issue.
I think your comment conveys your obliviousness of the problem domain.
The main driving need for prompt engineering is not an inability to "talk like a decent human being". That's just your personal need to insult and demean people who are interested in a problem domain you know nothing about.
The main driving need for prompt engineering is aspects like not being able to control how context is formed and persisted in a particular model, and how to form the necessary and sufficient context to get a model to output anything interesting. Some applications require computationally expensive and time-consuming runs, and knowing what inputs to provide to a system which by it's very mature is open-ended is a critical skill to adequately use the system in professional settings.
Let's put it like this: GitHub copilot is a LLM service which is extremely narrow in what are their applications and use cases. Yet, you can't even get it to add unit tests to a function following a specific style without putting the effort to build up the context it needs to output what you expect.
I was trying to understand what prompt engineering was, because I thought there is no way this is a discipline for how to ask ChatGPT questions... And then I realized it was...
Sure, I get that there is much to learn regarding formulating effective prompts, but a new career path?
until the next one, of course. ;)
for example, in rough order, some past hype trends: 3GLs, structured programming, initial AI (then AI winter), expert systems, CASE tools, 4GLs, OOP/OOAD, UML and round trip engineering, design patterns, dot com boom (and bust), agile, functional programming, Web 2.0, SaaS, crypto, Web 3.0, big data, data science, ML/AI.
most of them had or have some actual benefits, but nothing like the hype parrotted, by those with and without vested interests.
been there, seen them, from the third or fourth one onwards.
https://news.ycombinator.com/item?id=40957064
somewhat corroborates what I said above.
i counted, it's about 20 hype trends that i listed above (and don't forget that I may have missed some).
it is roughly 6 decades since the computer industry started, taking a start year of 1960.
so, 20 / 6 gives us an average rate of over 3 hype trends per decade !!!
about one every 3.3 years.
I myself would have thought it would be less often.
even if you make it 7 decades, 20 / 7 is nearly 3, so is still in the same ball park.
phew.
Remember Fifth Generation Computer Systems ?
https://en.wikipedia.org/wiki/Fifth_Generation_Computer_Syst...
I keep going back to the basics: Serverless is servers. Machine learning is servers. GenAI is servers. And, from what I've heard, most of AWS revenue is servers and storage.
(For the record: I am also an AWS Hero, and an AWS customer since 2006.)
The way that cloud businesses work, you sell the servers for about as cheap as you possibly can do. Instance prices are all a race to the bottom among the providers, because servers are largely commodity hardware that's easy to get from any number of providers, and it's one of the first prices customers see, and often plays a big role in their choice of provider.
So that's not where you make your profits. So you're right, lots of revenue, but crucially, there is no real profit. There never will be. That makes it a boring product, not worth focusing a lot on from a marketing perspective etc. Same tends to go for all of what you might think of as the basic building blocks of the cloud. e.g. object storage prices are often really close to what it actually costs to provide the service.
You make your profits on what you sell that runs on the cloud. All those additional things like databases, streaming services, kubernetes bits, functions etc. Those are where you make your actual profits. GenAI is a big potential profit driver for AWS, so that's where they're pushing. A couple of years ago it was "$foo, but on Kubernetes". Before that it was "$foo, but Serverless". They're just pushing where the profit and interest is, and pretty much always have done.
sort of side-note: Gartner's evaluation of cloud providers got really absurd around kubernetes stuff. Because one cloud would do it, you'd miss out on points if you didn't also add it, even if being on kubernetes literally added zero benefit, or arguably was worse. Same for "Serverless". It didn't matter if customers were actually using it, or wanted it, if AWS/Azure/GCP launched it, you'd better have it too.
The biggest cost for cloud storage services is not disks. Never has been. They're pretty cheap all things told. It's the per rack operational costs that dwarf them. The cost of electricity, cooling etc. You're also paying for the durability and accessibility that is built in to the software. It's why you see, e.g. backblaze obsessing about their server specifications and how many disks they can cram in to the server. Everyone is trying to maximise the rack data density, and trying to tow a really fine line on having just enough compute power in the individual server.
> It's the per rack operational costs that dwarf them. The cost of electricity, cooling etc
Ok lets do some simple math - lets say you have a few racks with 50PB in there, colocated. Power + floor space + remote hands will not cost you more to than 250k/y and that’s being very generous. Dividing by 50PB thats .5c per gig PER YEAR. Can you explain why GCS/S3 charges 50x that not including egress which is also ridiculous?
Also, it’s easy to beat most costs if you get consumer grade hardware and don’t refresh it more than once a decade. I see that often when folks compare costs, but that’s dishonest and just kind of shoots their credibility in the foot.
Private rates are cool except they come with strings attached and even discounted 90% its still not even close to break even on decent sized commits. Purely on costs the cloud cant win whichever way you slice it.
I don't think that's it. Those who migrated their EC2 apps to ECS/EKS/Fargate/App Runner have already migrated, so there is diminished returns in pushing those technologies.
The same goes for serverless. The whole world already adopted this to it's full extent. Those who want/can use these services, are already running these services, and AWS is already getting better utilization rates from their idle computational resources from this.
These are not fads. They are already infrastructure.
What we are seeing is additional high-level services being released to meet customer demand. There's now a massive need for training and running your custom private LLMs. There is absolutely no justification to skip the revenue you can generate by serving these markets.
I don't agree. I think AWS has always been extremely customer-focused, and they scramble to offer whatever service might have any traction at all from customers. It's just that they are already providing the low-level baseline services, and now they are progressing to offer increasingly higher-level ones.
I'm talking about machine learning-driven firewalls, backend for mobile applications, video streaming, edge computing, even Blockchain and now LLM services.
As much as it might surprise you, there is plenty of real-world demand for these services. You might accuse it of being "bandwagonny", but if you take an objective look at it you'll find that they are playing the role of supply store owners during the gold rush. It comes at no surprise that AWS is the one part of Amazon whose revenue is growing massively year-on-year, with the last report pointing to a 17% growth year over year.
https://www.cnbc.com/2024/04/30/aws-q1-earnings-report-2024....
Is that what you would call bandwagonny?
If nobody wants you to use them, is that because everyone already has as much conventional architecture as they need? Perhaps the new opportunities are all in AI because we've pushed conventional stuff as far as it could go, and we were just rearranging deck chairs.
I'll be honest that, if we've run out of ideas, I dunno if AI really solves any problems I want solved. But even if not I don't see how appealing to AWS fixes anything.
So, I do not belief that clouds are at the end of their innovation.
There's a ton of low hanging fruit in all the cloud vendor products. Look outside AWS at tailscale, vercel, and fly.io for some obvious examples.
> The same goes for feature releases. If the vast bulk of all new feature releases are geared towards GenAI, it implicitly means AWS is rerouting investments from classic infrastructure to shiny GenAI. It means that the products I love get smaller budgets. It means that the products I use will not get the next feature I want, or only at a slower pace.
I think the article does hyperbolize a bit, but this seems like a hard truth. Unless AWS has hired an entirely new swathe of AI-focused engineering talent, or if their public face at events is significantly disconnected from where they're spending their real money.
Why can you use a public HTTP gateway, but not in a VPC?
All this stuff would make my life so much better than any form of genAI.
AWS should be spending significant time explaining how their giant portfolio of conventional tools are improving. If they have stopped, they've lost focus. But hopefully it's just the marketing team focused on magic beans.
See but that won't work because leaders are dumb. The central tension of civilization is that the smart ones are needed down in the boiler/operating room to solve complex low-level problems, and whoever's left must steer the ship.
We're really fucked, aren't we?
It doesn’t cost them anything to re-route resources from the keep the lights on work, and has a potentially massive career impact if they’re the first ones to do something sensible with genAI.
Even if most of the work fails it’s still worth it. It’s to their benefit to hype AI beyond all reason because it justifies the resources they spend on it.
The problem being that nobody listens of course. You still have to build heaps of genAI crud because leadership is so excited about it.
Then you end up building a free text box to automatically determine one of 4 different problems the customer can have, and scratch your head wondering why we don’t just provide 4 options instead of letting the customer write a whole story...
However, what happened is that it became apparent that not everything needs to be big data. Business needs will shine through as they always have and dictate what is truly important.
I'm not afraid of the wave of gen AI. Think of it as the new power tool that just came out that everyone's currently talking about. You'll add it to your toolbox because you don't want to be obsolete. It'll blend into everything else once the hype wave is over.
Enhancements expected by the community will be delayed, engineers need to come up with temporary solutions which needs to be rewritten next year, new ways of solving problems will not be shared, new content will be created only for a couple of products.
The problem is, for all it's talk over the last few years, AWS remains a complete non-player in the GenAI space, much less so than Azure. In my opinion the problem is exactly the same as for every other high-level service they've tried to launch. QuickSight, Lex, Polly, Cognito, CodeGuru, SageMaker, etc: they're not good. Nobody ever said "I really like QuickSight, I sure wish it had GenAI capabilities". So when the hastily-expanded QuickSight team(s) then goes on to release 42 different Q enabled SKUs, nobody cares. For various reasons, AWS is organizationally incapable of launching a non-infrastructure product that is simply great, as doing so would take attention to detail and deeply caring about things like UX which are anathema to Amazon.
On the positive side, GenAI model access will be commoditized and part of the basic undifferentiated cloud infra, and AWS will do fine there.
I’d argue that nothing AWS launches is great at launch. The good products we actually use have all been around for 10+ years.
It was an awesome (and awesomely overwhelming) experience, but I completely agree with the author. GenAI EVERYWHERE.
The other topics that the author brought up from re:Invent 2022 were still present, but not without heavy mentions of how AI contributes to them.
That said, I have some predictions that might make OP happier.
DevOps and Platform Engineering is still a hot topic, especially in a world where companies are repatriating back to the data center (or are at least going hybrid). All of the 2010s bare metal tech (Foreman, Ansible, etc) are going to come back in big ways, and Kubernetes consumption will only increase. eBPF and systems engineering is still hot and will really help here for high-performance observability.
Companies that won't repatriate or want to use the cloud for prototyping will want to focus on cost optimization. This requires serious cloud engineering skills (using spot instances and S3 lifecycle policies is table stakes; much more can be done, especially on reporting and automation).
GenAI will help here (super helpful for analyzing time series data and surfacing patterns), but having the fundamentals will always be useful here.
c.f. Google IO keynote this year. I couldn't tell you a single thing Google is launching this year, beyond limited, rushed features where Gemini chat is in a side pane.
And that's not on me: it's because Google literally didn't talk about a single other thing.
And as usual, Google is out of touch and doesn't get the joke, c.f. at the end, Sundar presenting their own count of how many times they said AI.
I sorely miss tech industry of the 00s, I simply cannot imagine ex. 2000 Apple/Steve Jobs falling for this. There's this weird vapid MBA brain drain in charge everywhere. But hey, stonk goes up.
Most tech companies don't have a whole lot to show right now, so AI sucks up all the oxygen in the room. This becomes a feedback loop with the stock market, too.
> There's this weird vapid MBA brain drain in charge everywhere.
Yep. And Apple's playing along as well. Their latest WWDC presentation has the most weird tone I've ever seen in their presentations ever: "we added some AI features and they're pretty cool I guess...also it's super private! Here's all the ways it's kept private."
So much hedging going on. So little excitement. Because they're just playing to someone else's tune, and they're not good at doing that.
2024 Apple with AI: this is our best year ever. Look at our CTOs hair. We fixed Siri being eons behind, now if it thinks ChatGPT can help you can tap a confirmation dialogue.
I wouldnt be so quick to assume that. Let’s not forget that Jobs bought Siri and then integrated it into every platform they sold.
AI has been a buzzword for literally decades. It’s just exploding in popularity right now because the capabilities of GenAI have recently exploded.
It’s a little like how VR and AR has been around for decades. It’s just taken this long for the technology to make mixed reality a possibility for the masses.
No he would not entertain any of this nonsense. IIRC he had a hard time with the Siri demo too. It appears Apple is happy to take itself and shareholders on a side quest with all their cash…good time to be earning interest on it too.
Haha be kind to your ex-employer. This was supposed to be a joke. I agree they don't get it but they at least tried this time. I ranked this high on the Sundar-joke ladder (which is a low bar I know).
For example, the team / leadership / foundation behind Home Assistant has been pushing AI features hard in the past 18 months or so. This coincides with my feeling that there hasn't been any relevant improvement in Home Assistant's core features and usability — it's in stagnation for over a year now.
This is of course my own opinion, but it makes sense: if a significant share of resources is spent on AI stuff, that share is not available anymore for other needs.
> Your first leadership principle is customer obsession: “Leaders start with the customer and work backwards”. > I’m your customer, and I’m begging you: please let me be a cloud engineer again.
However, as with many enterprise products, the author is not the customer; it’s the user.
The customers are the companies that buy AWS because it’s an essential technology for their strategy. When the whole tech world is talking about generative AI, they want to be there, and Azure seems to be ahead because of the MS deal with OpenAI. (even if they are not ahead, customers' perception matters most).
So basically, what Amazon is trying to do by making all of these conferences and announcements about GenAI is to send a message to their customers: we are ahead on the wave of GenAI and you can still trust that our products are going to help you be on the hype.
> I’m your customer, and I’m begging you: please let me be a cloud engineer again.
Only AWS knows how many H100 GPUs they have, how busy they are. How many people are paying for them, how many people want them and can't get them, and how many people just don't care at all.
It's possible that the focus on GenAI for Re:Invent 2023 wasn't based on any hard data like that, and is really just up to the whims of Adam Selipsky since Jassy moved over, but maybe someone who better knows their planning process can comment.
This opinion is based on admittedly anecdotal experience, but I’ve worked in a large range of domains on AWS over the years and by far the biggest AWS bills were for startups specialising in GenAI.
Plus often businesses need GenAI to be trained on their own IP (ie stuff sensitive to their own business that they don’t want in the public domain).
—-
Point 1 will be decided over the next few years as creators take companies to court (or “ethical AI” starts to displace the current models trained on unlicensed content)
Point 2 cannot be resolved without training your own models.
——
Let’s also not forget that LLMs are just one part of the GenAI movement. There’s audio and image generation too (Plus video, but that’s more an extension of image). In fact it was the latter that I worked on.
And then you have other areas of AI outside of the generative space too. From hundreds of different applications of image recognition to sound processing to searching for other kinds of bespoke patterns. These are all areas I’ve worked in too.
Often a GenAI product will require multiple different “AIs” to function, as part of a larger pipeline that appears like a single opaque box to the customer. And most of those models in the pipeline likely aren’t generative, let alone LLMs.
Startups doing machine learning on EC2 might have large bills, but that isn't what the AWS focus on generative AI is about!
Stuff like SageMaker et al still require compute, storage, network traffic, etc.
For example, some instance types are only available on SageMaker.
AWS aren’t amateurs, they know how to promote their platform effective.
Ironically, trying to compete like that, and then focusing on that, causes the problem they're trying to avoid IMHO. They're always going to be third as long as MS gets to serve OpenAI and Google has in-house AI talent building models that are top-tier competitive. And if you set this one farking thing aside, they're #1.
* ex. first time I've seen "FM" as an acronym is on the page you link. they mean "foundation model", which itself is a term I'm likely to see in the Economist, but never on HN. Colloquially, it means "big AI like ChatGPT"*
In private they are truly thirsty for AI applications they can write uses cases on that they even offer upwards of 100K credits for Gen AI purposes only.
I think the technical specialty that will be most at threat from automation by AI would be the exact job that he authored has -- solutions engineers that build commodity cloud infra on AWS, Azure, G cloud, etc.
Look at progressions and range of abstraction between standard sys admin IT work to serverless deployments, especially with IaaC tools.
You can describe your architecture to chatGPT and it can spit out a CloudFormation YAML. It will be rudimentary and poor, but I could see a Gen AI tool offered by cloud providers where al you do is describe your app and then deployed infra on your behalf, and optimize form there.
Not trying to talk down on folks who do this type of work, but sharing my opinion on where I think the author is ultimately coming from.
Can you point to any actual AI product in this space that functions? Everything I've seen is like, if you squint then it kinda looks like it's doing something, but it's actually producing something embarrassingly wrong, unsafe, or otherwise unusable. And no, having a SME repeatedly prompt until it does the right thing doesn't really make sense.
If we're just talking about hypothetical tools that someone could make, but haven't, we're talking about magic.
Today? No. But I think we will get here sooner than automating on any other type of engineering role.
You might know the difference as an SME, but if you're not, and it passes terraform apply, it's getting used.
So that’s not nothing.