538 karma · joined August 28, 2024
When you start a project not everything is DRY, and you don't start pulling out shared helpers until they're called for. If you're a slacker and expect the agent to magically fix everything with one-off prompting and only moving forward, it's not going to work. Agents prefer to carry existing patterns forward, just like many humans.
The shared helper example from the article requires intent to refactor. If you ask the agent look for smells and refactor, it will happily assist you with that. If you ask it to add a feature, which happens to add duplication, you get a feature with duplication because you didn't specify anything else.
Better yet, add tooling like static code analyzers (shout out to Credo) to your development pipeline and catch stuff like this regardless of LLM. Verify against a mechanical baseline, with LLM judgement layered on top as-needed.
"The most frustrating part: I thought I was outsourcing maintenance to the LLM, but the slippery slope I found myself on was actually training it to have ever-worsening habits."
That's on you. The LLM is a tool and you're not using it well. Build a good foundation for the agents to anchor on, and you'll get good results moving forward.
I think this sums up the problems pointed out by the article, as well as most organizational problems in general.
When I was younger, I thought management was basically useless, do-nothing mucky mucks, who did more harm than good. This can still be true, but I've changed my opinion. Hiring and firing is hard. Creating a strong and healthy culture is hard. Defining success, and steering the ship to it is hard.
I think as long as money is the measurement of success, particularly from a short-term viewpoint, nothing can change. Public companies basically cannot evolve their form, because boards and shareholders will rarely agree to leave money on the table for an opaque long view that prioritizes culture over profit.
I hope it's as good as it sounds. Thanks for open sourcing <3
The only way I've found to really force rules is via hooks, and even then I think it's just prompt injection? Maybe some kind of hook/checksum thing to ensure you're running unchanged tests, but as you're pointing out, the agents can get sneaky and do weird stuff if they have write ability.
> validating that the LLM didn't disable tests it didn't agree with
Provide a test runner and force the agent to call it. Have it emit something if you want evidence.
This is the way. If I'm traveling I consider Tripadvisor's top lists as things to avoid, and don't use the site otherwise. Bangkok in particular, you can basically walk blindfolded and you'll run into amazing food.
I'm interested in quality, but I didn't find these notes enlightening, and couldn't even finish the article.
It wasn't that long ago...
It's pretty easy to see that most things above plankton aren't rote automatons. Anyone claiming otherwise just has an agenda to sell.
No. Lighting up the night is an abomination. Reflect Orbital can go fuck themselves.
Just waving the flag for team-oat here. Thanks Canadian oat farmers!
I don't actually want Claude to have those memories, but they often point to gaps in my harness. I'll occasionally sweep the memories, pick out what should go into the harness or CLAUDE.md, then delete the memories.