Hacker News Reader

> AIs when writing code introduce vulnerabilities at a rate similar to humans writing the same code.

AI regurgitating all the insecure code AI companies scraped from stack overflow and github isn't going to give you something too different from what the humans who put it there in the first place came up with. Garbage in, garbage with random hallucinations out.

View on HN

This is simplistic to the point of being blatantly wrong. Training data isn't garbage. It's programs that do their job, but that are sprinkled with errors. Uncorrelated errors gets averaged out during autoregressive pretraining. Correlated errors can be somewhat suppressed during post-training. Hallucinations (of the generalization-error kind) can be dealt with using synthetic data that improves the model's generalization.

So with all those mitigations why do LLMs keep writing insecure code?

Provable correctness is much harder than focusing on the happy path. If a pipeline doesn't include a look-for-the-vulnerabilities stage to save costs, a model wouldn't go out of its way to do it. The models are trained to do what they are asked to do.

I forget to add the obvious: some garbage gets through despite the mitigations. In the limit of pure garbage input, you'll get a model that internalized garbage generation. And you'd be better off throwing it away and starting over.

Reply on news.ycombinator.com