Hacker News Reader

Comment by kees99 | Hacker News Reader

> model to add new vulnerabilities to your code so it can present you with more exciting findings.

Not so sure about this level of 4D chess capability just yet. The other day I asked Opus to come up with some cleverly vulnerable crypto code "as a good, hard challenge for an IT security student", and results were quite mediocre.

And by mediocre results I mean that 3 "cheap" models out of 3: qwen3.8-27b, glimmer, and luna - all were able to find every problem planted there, with fairly little steering, and no spoilers.

View on HN

Let it iterate, give it access to the cheap models, and tell it part of the requirements is that the cheap models shouldn't be able to solve it with such and such a prompt. I expect it will be able to zoom in on something.

One shot generating a problem of exactly the difficulty the user has in mind is a very difficult problem for anyone. "IT security students" span a wide range of capabilities, but I would expect most of them are worse than qwen3.8-2.7b at this kind of work.

To use a motor analogy: A lot of people just run these things on idle and never realize that you can put a load and apply power.

Give it a goal, a test for the goal, a harness to iterate in, and tell it to iterate. It'll top out a wee bit higher.

Yes, you asked it for a good, hard challenge for an IT security student. I'd suggest checking if it succeeded by asking an IT security student, and not an LLM.

Students are still learning, and have not been trained on vulerabilities. Qwen, glimmer, and luna have been. LLMs are largely better than IT security students at finding vulnerabilities. In many aspects, they're better than IT security experts at finding vulnerabilities.

Train it on human content, get human emulating behaviour, including all the perverse incentives and bad acting we see from people, fictional or otherwise.

  > all were able to find every problem planted there, with fairly little steering, and no spoilers.

Given that LLMs prefer output of LLMs (of the same LLM and of others) [1], can it be the case that they generate "hard challenges" from the manifold of challenges solvable by (other) LLMs?

[1] https://arxiv.org/abs/2404.13076

I would worry more about 5D chess or 6D and even 7D chess. This article talks about teaching models to be good and methods that could train goodness, but it sounds like there are some unsolved problems to graduate to 8D+ chess.

Reply on news.ycombinator.com