I’ve also seen a lot of issues with co-workers using an LLM to write their readme files. I look at the readme for what return values I should get, go to use them, and get an error. I check the code, and sure enough, none of the variables in the readme exist. The LLM just through they sounded good. Things like this I would say are pretty objectively wrong.
I remember seeing this maybe 6+ months ago, but using paid plans, RAG, and a high thinking mode has eliminated a ton (almost all) of those kinds of hallucinations. Open models and free tiers are not there yet though.
> I’ve also seen a lot of issues with co-workers using an LLM to write their readme files. I look at the readme for what return values I should get, go to use them, and get an error. I check the code, and sure enough, none of the variables in the readme exist. The LLM just through they sounded good. Things like this I would say are pretty objectively wrong.
LLMs don't co-sign the quality of PRs though — your coworkers do. It's not unusual for docs to get oudated and not be maintained enough in small codebases, but that's not an LLM specific problem.