Comment by alexpotato | Hacker News Reader

Comment by alexpotato | Hacker News Reader

I work in DevOps at a firm that has been very enthusiastic about using LLMs (in the good sense).

The phases were basically:

- try out having the LLM do "a lot"

- now even more

- now run multiple agents

- back to single agents but have the agents build tools

- tools that are deterministic AND usable by both the humans (EDIT: and the LLMs)

The reasons:

1. Deterministic tools (for both deployments and testing) get you a binary answer and it's repeatable

2. In the event of an outage, you can always fall back to the tool that a human can run

3. It's faster. A quick script can run in <30 seconds but "confabulating" always seemed to take 2-3 minutes.

Really, we are back to this article: https://spawn-queue.acm.org/doi/10.1145/3194653.3197520 aka "make a list of tasks, write scripts for each task, combine the scripts into functions, functions become a system"

I've a bunch of technical, but non-engineering types around me, and a few of us engineers keep banging the gong on the fact that they can't trust the output of an LLM. That the best way to leverage AI is to get it to write the code in whatever language they prefer, so they get a simple and repeatable tool out the other side. In many regards it's a liberating tool when used like that. I've got TPMs that are really able to use it as a force multiplier for themselves, building small tools that help them, without having to tie up engineers to produce it.

In numerous cases, though, there are folks asking it to go interrogate some stuff they've set up MCPs for, and produce reports from it. If you do that it will give you a different answer every time, even from exactly the same input (because that's how LLMs work) and you just can't guarantee that any of them are accurate. It's a probabilistic layer, and the reports you need to generate need to be deterministic.

The problem is we're so accustomed to the deterministic nature of the large majority of the software we work with. The output is plausible, too, which only exasperates the problem. Folks just assume it's correct.

Sounds like the bleeding edge.

LLMs are tools, but unreliable

They can magnify the reach of a person, but not replace them

Having LLMs write the tools is the correct approach for magnifying the reach of a Dev Ops programmer

Reply on news.ycombinator.com