AI and Infrastructure Engineering | Hacker News Reader

AI and Infrastructure Engineering

omegion.dev

72 points0megion41 comments

> The honest tradeoff

> That’s not a hypothetical cost - it’s

> But that’s exactly the

These are strong tells for an AI authored article. I know we're all tired of people saying "this is AI" as it's harmful to discussing the content of an article, but articles about AI, (seemingly!) written by AI, extolling the virtues of AI -- don't really seem to drive the discussion forward to me either.

Maybe I should have added this part. As a non-native English speaker, using AI to write RFCs, ADRs, and internal documentation, my own writing style shifted so much to the direction that my writing seems like AI.

Weird, I put your blog post through Pangram Labs. It said 96% of the article was written wholesale by an AI.

So, if you "write like AI" like you claim in a later comment, you are so good at it to accidently hit Pangram's extremely low false positive rate.

It is trivial to check this stuff. Just be honest.

I am also a non-native English speaker but that doesn't mean you start speaking Claudian English just because you interact with AI.

If people do not like that particular style of writing and you understand it clearly, why would you double down on it? Especially if your goal is to share your writing with the intention of them reading it?

Furthermore, why not just blog in your native langauge if that is comfortable for you?

That is fair. However, the fact that we have this discussion because I wrote the post in English, I wouldn’t reach out to my fellow engineers to discuss the matter this broadly.

I don't think the existence of this discussion has anything to do with the topic we are discussing. "I wrote this myself but I only know how to write like an AI because that's how I talk in English all the time" is a weird stance to take. Furthermore, your comments are a completely different style of writing so how are you able to write like that here but suddenly switch to Claudian English for the blog?

Regardless, my feedback to you would be to disclose AI writing if you do use it (in the blog itself, not as a reply when called out for it) or if you genuinely write like that, aspire to be better than AI generated content if you care about what you are reaching out with in the first place imo.

Jeez, give it a rest. He's made a good point about accessibility.

I guess you don't like it, so skip it then? Don't constantly relitigate.

Genuinely hilarious copout. "It's not AI, I just sound like one!!!"

In that case, sorry for getting it wrong and throwing shade.

I noticed the same on myself.

In a few years it would be interesting to see a research about how LLMs influenced the choice of words in the general population.

I’m one of the people who regularly complains about AI slop posted here.

But fwiw, your post didn’t trigger my LLM detector, and you’re writing thoughtfully about things that many of us are thinking about. Kudos.

The rest of the blog post doesn't strike me as AI generated. Hard to tell, and personally I think "hard to tell" is about as good as it's going to get from here on out, so I give the author the benefit of my doubt.

Agree that using AI for writing is a habit we collectively need to drop immediately.

I always check out the content whenever I see accusations of AI writing because I feel people can be a bit too suspicious these days but yes, this is as close to "AI but i changed how the em dashes look" writing:

Your idea that the AI is writing articles saying AI is good is very funny though

It's frustrating to find AI so prevalent in writing. It takes some nugget of a human written idea, dilutes it and increases word count, only to make the actual core idea less obvious (or ultimately summarized by someone using AI).

Poor hyperscalers have got to make money somehow...

> Four years ago I hand-wrote a nested for loop - four levels deep, tagging subnets across regions and availability zones in another AWS account - and it took me about an hour to get the syntax right

I do believe this kind of engineer requires AI. I also hope I don't have to work in the same teams as them. Seasoning teaches you to make changes in other parts of the codebase rather than do this kind of data mangling on local variables, which is incredibly hard to troubleshoot and very brittle - not to mention it's a nightmare to read back and understand.

It is Terraform’s configuration language. Locals mean something specific : non-exported variables. Constructing the map or whatever prior to plan phase is pretty idiomatic. Note the key in the map is a string concatenation. Not something you would do elsewhere but a constraint of that framework.

I've been using Terraform since it came out. I know what it is.

Yes, your post is more an indictment of Terraform itself, rather than the grandparent's usage of it.

I've had pretty good luck with Crossplane, it has its warts, but it removes the lack of obviousness that accompanies Terraform.

Unfortunately, reconciling state against infrastructure configuration seems to be a pretty tough problem to get right so no solution is amazing. But being as declarative as possible upfront and not obfuscating it behind a build time execution phase seems to help AI reason about it.

I've looked into Crossplane, and I like a lot of things about it. Unfortunately when we spiked it, the providers we required were a major let down.

In terms of these horrendous HCL constructs, I'm not going to pretend I haven't written them myself ever, but every single time it's self-inflicted and can be solved by redesigning the data types. As with any other tool and language, if you design your codebase for its weaknesses, you'll have a much better time.

Nowadays all providers are, or can be somewhat easily, just use Upjet to convert a Terraform provider. They walked away from the custom provider maintenance model awhile back due to its lack of scaling potential and loss of network effects from Terraform being the de-jure framework for infrastructure API's

I will say that yes, quite a few of the providers are half assed and have bugs. Mostly around how reconciliation works, but also gaps in the provider in how it writes fields due to some Terraform idiosyncracy. I often end up having to ignore various fields in the yaml to prevent drift. But now you can wrap your own without too much effort.

HCL itself definitely footguns you into it, it's perverse, because it's intended to be a mere configuration DSL and yet it has since its inception been square pegged into a round hole to do imperative language construct things that it was never designed for. Which is why you get insane practices like "never use an array to iterate over infrastructure in a loop, because removing an element forces a recreation of every resource after that loop because the position of the element in the array shifted to the left". These sort things are completely logical things you would do in a not insane framework (who doesn't use an array to loop over things in a normal software language?), but are a well-known landmine with HCL and Terraform.

You could say it was a mistake to use a language like HCL for Terraform and I would agree. It's a half baked idea that became production way too fast. Reminds me of Javascript.

Good to know about Crossplane, thanks!

I agree with your points around HCL. This is one of the reasons I liked Chef's DSL - it was Ruby. It was foolish to do too much Ruby stuff around the DSL, but if you kept it to the right amount it would help immensely, so the DSL never needed to be bent out of shape for each edge case and extra functionality.

Leave iterations and conditionals to the languages that are good for it.

I would not be quick to judge the person without knowing

- the kind of work environment they were in

- the quality of the existing code base

Another shameful quote from the article:

> Kubernetes made server management easy enough that we stopped building our own node images at all - we just use whatever the cloud provider hands us, an AWS AMI built for us, no questions asked.

This dude is just bad at his job.

SRE here, why would you create your own node images for Kubernetes Nodes? Unless they deliver some weird kernel that would cause containers not to run, there is no reason not to take cloud provider image.

If you need something extra on the nodes, 99% of the time, DaemonSets will do the job.

In many spaces, higher security requirements.

> Did Kubernetes kill Ansible? Kind of. I haven’t written an Ansible playbook in years - if you handed me one right now I’d be squinting at the module syntax like I’d never seen it before - not because configuration management stopped mattering, but because Kubernetes made server management easy enough that we stopped building our own node images at all - we just use whatever the cloud provider hands us, an AWS AMI built for us, no questions asked. And when did I last SSH into a node to debug something?

K8s did not kill Ansible because of ease. Ansible is obsolete because we moved from creating a machine and changing it to instantiating it with the changes we wanted. Ansible makes sense when you are continuously updating a server-in-place and is decidedly less useful when you spin up a new unit with the new configuration when changes are made. Even in this instance the underlying skills remain the same.

For a lot of people in practice, Kubernetes was what replaced use of Ansible (or Chef etc.), so the OP take makes sense.

Also, the transition you describe isn’t the whole story. Containers meant that the node configuration essentially stopped mattering - the node contains no application-specific code or dependencies. This shift wasn’t due to Kubernetes alone, but Kubernetes became the standard way to manage such systems.

Matches my experience working with LLMs on Kubernetes. They are amazing at writing the boilerplate yaml, and fantastic at using kubectl to monitor and debug clusters (with guardrails and readonly permissions in place of course.) if you have a staging/test cluster, you can get an extremely fast feedback loop if you let it apply changes as well.

It’s a good example of how highly structured, well- documented, opinionated frameworks help LLMs do well. There’s just less room for subjective issues when there’s more structure around it.

This is an extremely common misunderstanding about AI in software. In order for AI to "move up one layer of abstraction" it has to actually be an abstraction. But AI output will _always_ be flawed in ways that mean you still need to fully understand the layer below it.

I’ve been in engineering long enough to know it’s cyclical: the new thing always eats layers that demanded expertise until it demands expertise, at which point something new tries eating that layer. It was shell scripting until it became Ansible, until it became HCL, until it became K8s, until Helm, until AGENTS.md, until…well, until we wound up back at building images again for local compute because cloud costs are spiraling upward and privacy/sovereignty are the focus again.

It’s kind of ironic that, for all the expertise I’ve cultivated, all the skills I’ve grown, all the knowledge I’ve gained, I’ve found myself back at the start of my career cycle again in terms of manually defining and building OS images for virtualization on local hardware. AI has not been much help in this regard because of the hundreds of OS settings I have to understand and set according to security posture - I’ve been unable to get Claude/Qwen/etc to build an effective hardening profile for something as fundamental as CIS2, because it always presumes an AD environment on a CSP as opposed to a local VM. Believe me, I would love for it to do so, because this is dry, uninteresting work, but I’ve spent more time wrangling AI than I would’ve spent writing my own Powershell.

AI is, at least in my realm, a neat tool for basic, repetitive, boilerplate work. Very much a sudo make me a sandwich type of deal, except for YAML. That’s not me knocking the tool, just my two cents on what function it serves at present. Put another way: we wouldn’t have made our sixth hire a Support staffer if AI could do more.

This works because he has decades of working experience at the foundational layers and understands each layer of abstraction, what it solves and the interaction between the abstractions. Can still reason it out, if not the fastest at troubleshooting lowest layers because its no longer in muscle memory. AI can make you productive if you have the necessary experience (which takes a long time).

But, AI natives have none of the knowledge of ANY layers, even at the highest layer at which they will be using AI. They only learn how to do prompts, etc and nothing else about what is being done. This is just like anyone can be a pilot because planes mostly fly themselves.

This article basically captures my current professional life to a high degree of accuracy. I still enjoy the work. I still get the satisfaction of conceiving a thing and seeing it realized. That has always been at the root of the pleasure that I get from this job. Like the author I find myself drifting away from the underlying layers that the agents has so thoroughly mastered, and like the author I wonder about its eventual mastery of the parts I still consider my domain.

It’s a good example of how highly structured, well- documented, opinionated frameworks help LLMs do well.

For most users, Kubernetes has added complexity and cost, but not value. The sooner we realize this, the sooner we start thinking about how things should be built. Whether or not you have LLMs has nothing to do with that. The LLM will happily help build your crap even faster!

Maybe people will finally realize they need to start rethinking infrastructure automation. I am.

> For most users, Kubernetes has added complexity and cost, but not value.

This may be true for small, simple systems, such as basic web apps, that don’t really need Kubernetes. If your system can fit on a single machine, or even two or three, why would you use a cluster orchestrator to run it?

But if you’re at a real company - not a mom and pop shop or similar - and you have multiple services implemented by multiple teams, you’d be hard-pressed today to find something that makes more sense than Kubernetes. Yes, there are PaaS offerings that can hide some of the inherent complexity, but they tend to run into limits, cost issues, etc. quite quickly. Not to mention they’re often implemented on top of Kubernetes anyway.

Kubernetes is worth learning if you want to understand the needs and design of non-trivial software systems. Everything it offers corresponds to requirements such systems have, and it provides an extremely well organized, unified way of managing that. It’s one of the most well-designed software products of the 21st century.

I think the issue is that there is a not yet emerged standard for the mid size companies that we hear less about here on HN but make up a huge amount of business in the US. Kubernetes may be the gold standard for larger companies or tech heavy companies of all sizes. But what about the mid size organisations ?

I've worked for a mid sized and a large company in the same fintech space. Like you said larger one use eks and the entire aws stack, mid sized one is still using 15 years old on prem stack and partially self hosted cloud service.

Now they are moving to microservice fleet but will have so much devop overhead, so I'm building this platform that's basically a simple managed microservice + job service, not sure how many companies will be interested

Mid-size orgs can use PaaS if they want to. But the core issue is, standing up a system with high availability, security, automated deployments, etc. etc. is non-trivial.

There's a small but vocal subset of the HN audience who thinks "I can do all that with one hand tied behind my back," but they're not a scalable solution to actual business requirements.

I've worked at companies with 5 people (3 of them cofounders lol), with 30 people, with 100+ people, with 500 people, and with tens of thousands of people (global financial, healthcare, and telecoms corps.) Give me an example of these "mid size companies" that don't have a good technological solution available to them and I'll either tell you what they should be using, or why what they imagine they want doesn't exist.

MicroVM !

this post is so relatable, especially the locals where tags are four levels deep lol

Reply on news.ycombinator.com