Ask HN: What are your worst war stories bringing agentic applications into prod | Hacker News Reader

Ask HN: What are your worst war stories bringing agentic applications into prod

12 pointsyaoke2597 comments

For a bit of context, I’m currently creating a team of AI agents at work to generate reports by fanning out into a large amount of subagents to process a large amount of transcript data. When the analysis fails mid-way because of some individual step like an API call returns an error or the machine is out of memory, it would create cascading errors that break the entire generation with almost no visibility. I’ve just spent the past month rewriting the individual jobs as durable execution jobs on DBOS but just wondering if there are better solutions out there and if others encountered similar issues? And then there is the issue to reflect back the progress to the users which I’ve just been coding ad-hoc honestly…

When an agent fails at step 9 of 12, how do you handle that?

Roughly how many engineer-weeks have you sunk into agent infrastructure (durability, monitoring, human-in-the-loop, live UI) vs. the actual agent logic? Curious if my ratio is normal.

For those who built this stuff in-house: was it ever a build-vs-buy conversation? What would a tool have had to do for you to buy instead of build?

Do you currently pay for anything in your agent stack (LangSmith, Temporal, Braintrust, etc.)? What made that one worth a line item when others weren't and should I look into it too?

Be careful about the culture. Higher management had a massive panic because "AI stopped working" in a product we make. They started an all-hands meeting. Everything worked except the AI responses. Turns out we just ran out of credits. The bad part is we told everybody we would run out of credits in less that 3 days. Nobody cared until we had production outage. Now procurement is complaining about token usage.

It wasn't a crash that cost us the most time; it was a stale read. Two agents shared a plan file. One updated the file during the run and the other continued to work off the version that had been loaded at the start. Both produced plausible output. Nothing errored. We only discovered this issue during the review process, after wasting hours of generation time. I now treat any artefact that two agents can both access as a coordination problem, not a storage one. The cheapest and most effective solution was to stamp a version on the artefact and re-check it before acting, instead of trusting the copy read at the start of the run.

Definitely similar experience, building the "boring infra stuff" around it is a time-sink no one prepares you for...

what have you guys tried?

"war stories" buddy this ain't a warzone. so cringe

sounds like busy work you give to someone you dont expect to be here in 6 months.

lol I was hired to build agents. It's all some companies can think and/or dream about doing right now. This is a F100 company too. The feeling is "how can we turn everything into some sort of AI tool, but hey lets also change the scope constantly and use this week's #latest technology to do so".

Reply on news.ycombinator.com