Hacker News Reader

Comment by prettygood | Hacker News Reader

I'm so happy someone else says this, because I'm doing exactly the same. I tried to use agent mode in vs code and the output was still bad. You read simple things like: "We use it to write tests". I gave it a very simple repository, said to write tests, and the result wasn't usable at all. Really wonder if I'm doing it wrong.

View on HN

I’m not particularly proAI but I struggle with the mentality some engineers seem to apply to trying.

If you read someone say “I don’t know what’s the big deal with vim, I ran it and pressed some keys and it didn’t write text at all” they’d be mocked for it.

But with these tools there seems to be an attitude of “if I don’t get results straight away it’s bad”. Why the difference?

There isn't a bunch of managers metaphorically asking people if they're using vim enough, and not so many blog posts proclaiming vim as the only future for building software

I’d argue that, if we accept that AI is relevant enough to at least be worth checking, then dismissing it with minimal effort is just as bad as mindlessly hyping the tech.

You must be new here. "I use vim between", "you don't use vim, you use Visual Studio, your opinion doesn't count" is a thing in programming circles.

Internet commenters, sure.

It never broke into the workplace like measuring AI use among your employees. Nobody's asked me about how I've used vim keybinds to improve the company's growth in a performance review.

Well one could say that since it's AI, AI should be able to tell us what we're doing wrong. No?

AI is supposed to make our work easier.

What you are doing wrong in respect to what? If you ask for A, how would any system know that you actually wanted to ask for B?

Honestly IMO it's more that I ask for A, but don't strongly enough discourage B then I get both A, B and maybe C, generally implemented poorly. The base systems need to have more focus and doubt built in before they'll be truely useful for things aside from a greenfield apps or generating maintainable code.

Certainly, every tool is supposed to make our work easier or more productive, but that doesn't mean that every tool is intuitive or easy to learn to use effectively or even to use it at all.

Certainly, but aren't AI tools supposed to be intuitive and easy to use because we can communicate with them in natural language?

With VIM or Emacs I am supposed to know what Ctrl-X does. But with AI tools (ideally) I should be able to ask AI (in English) to edit the document for me?

Maybe the reason we can't do it that way is that, "We're not there yet"?

I agree to a degree, but I am in that camp. I subscribe to alphasignal, and every morning there are 3 new agent tools, and two new features, and a new agentic approach, and I am left wondering, where is the production stuff?

So just like in the JavaScript world?

I don't understand how to get even bad results. Or any results at all. I'm at a level where I'm going "This can't just be me not having read the manual".

I get the same change applied multiple times, the agent having some absurd method of applying changes that conflict with what I say it like some git merge from hell and so on. I can't get it to understand even the simplest of contexts etc.

It's not really that the code it writes might not work. I just can't get past the actual tool use. In fact, I don't think I'm even at the stage where the AI output is even the problem yet.

> I'm at a level where I'm going "This can't just be me not having read the manual".

Sure it can, because nobody is reading manuals anymore :).

It's an interesting exercise to try: take your favorite tool you use often (that isn't some recent webshit, devoid of any documentation), find a manual (not a man page), and read it cover to cover. Say, GDB or Emacs or even coreutils. It's surprising just how much powerful features good software tools have, and how much you'll learn in short time, that most software people don't know is possible (or worse, decry as "too much complexity") just because they couldn't be arsed to read some documentation.

> I just can't get past the actual tool use. In fact, I don't think I'm even at the stage where the AI output is even the problem yet.

The tools are a problem because they're new and a moving target. They're both dead simple and somehow complex around the edges. AI, too, is tricky to work, particularly when people aren't used to communicating clearly. There's a lot of surprising problems (such as "absurd method of applying changes") that come from the fact that AI is solving a very broad class of problems, everywhere at the same time, by virtue of being a general tool. Still needs a bit of and-holding if your project/conventions stray away from what's obvious or popular in particular domain. But it's getting easier and easier as months go by.

FWIW, I too haven't developed a proper agentic workflow with CLI tools for myself just yet; depending on the project, I either get stellar results or garbage. But I recognize this is only a matter of time investment: I didn't have much time to set aside and do it properly.

>I don't understand how to get even bad results. Or any results at all. I'm at a level where I'm going "This can't just be me not having read the manual".

>I get the same change applied multiple times, the agent having some absurd method of applying changes that conflict with what I say it like some git merge from hell and so on. I can't get it to understand even the simplest of contexts etc.

That is weird. results have a ton of variation, but not that much.

Say you get a claude subscription, point it to a relatively self contained file in your project, hand it the command to run relevant tests, and tell it to find quick win refactoring opportunities, making sure that the business outcome of the tests is maintained even if mocks need to change.

You should get relevant suggestions for refactoring, you should be able to have the changes applied reasonably, you should have the tests passing after some iterations of running and fixing by itself. At most you might need to check that it doesn't cheat by getting a false positive in a test or something similar.

Is such an exercise not working for you? I'm genuinely curious.

Some people shouldn't just be engineers in the first place, I guess.

You didn't actually just say "write tests" though right? What was the actual prompt you used?

I feel like that matters more than the tooling at this point.

I can't really understand letting LLMs decide what to test or not, they seem to completely miss the boat when it comes to testing. Half of them are useless because they duplicate what they test, and the other half doesn't test what they should be testing. So many shortcuts, and LLMs require A LOT of hand-holding when writing tests, more so than other code I'd wager.

There are a lot of comments on HN and other places breathlessly gushing about agents totally doing everything end to end, so I couldn't blame someone new to this space for naively assuming that agents would be able to handle a well-bounded problem such as test coverage reasonably well.

> naively assuming that agents would be able to handle a well-bounded problem such as test coverage reasonably well.

We haven't figured out a way for humans to do that well :P I still see people arguing about "80% test coverage is obviously better than 70%" and similar dumb sentiments that completely misses the point.

But agree with the first part, LLMs are massively oversold and it's hard to blame users for believing them. Tempered expectations as always win.

No, that was an exaggeration. The prompt was decent. I explained the point of the repository, that I wanted full coverage with tests, that it could keep going until it worked. Maybe that was still not enough. With how others talk about it, I must be missing something.

For tests, you need to be precise about what it should test, how it should test it, and what the assertions should be, otherwise you'll mostly get trash, they're exceptionally horrible at writing tests. Which makes sense, most programmers are too, but given the importance of correct tests, it's probably the part that needs to most human handholding right now.

“Write tests“ may not be enough; provide it with a test harness, and instruct it to “write tests until they pass “. Next would be “your feature isn’t complete without N% coverage”. These require the ‘agentic’ piece, which is at its simplest some prompts run in a loop until an exit condition is met.

> I gave it a very simple repository, said to write tests, and the result wasn't usable at all. Really wonder if I'm doing it wrong.

I think so. The humans should be writing the spec. The AI can then (try to) make the tests pass.

No, you have similar experience as a lot of people have.

LLMs just fail (hallucinate) in less known fields of expertise.

Funny: Today I have asked Claude to give me syntax how to run Claude Code. And its answer was totally wrong :) So you go to documentation… and its parts are obsolete as well.

LLM development is in style “move fast and break things”.

So in few years there will be so many repos with gibberish code because “everybody is coder now” even basketball players or taxi drivers (no offense, ofc, just an example).

It is like giving F1 car to me :)

you need to write a test suite to check his test generation (soft /s)

Reply on news.ycombinator.com