Hacker News Reader

Comment by TuxSH | Hacker News Reader

I find this - or perhaps the title - a bit surprising.

I've benchmarked GLM 5.3 and DSv4.1-F on my fully-annotated decomp of the Nintendo 3DS's kernel, which I have a good mental understanding of, tasking them to find vulns and other bugs (in Max mode w/ subagents). GLM 5.3 founds almost all the vulns in 30min for $22, while DS only found one vuln for $2 in 40min.

Perhaps DS works better where targets have low-hanging fruits than can be found fast?

View on HN

Makes sense, GLM 5.1/5.2/5.3 is a lot better than DS v4.1, I found the same results in other domains.

Given how fast and cheap DS is, it's just an ideal model with enough "IQ" to let it loose. Another thing they left out of the article, DS becomes really good if you provide custom tools for the task, on it's own it's mediocre.

How does GLM 5.3 Flash rank against it's big brother and the latest Deepseek?

Both these models are not bad but you can not trust them to make good decisions. I meant GLM 5.1/5.2/5.3 in the previous answer, not the flash version

Worth saying that GLM-5.3 isn't GLM-5.3-Flash's "big brother" the way one might think. GLM-5.3-Flash is not GLM-5.3 scaled down. While GLM-5.3 is based on GLM-5.2, and "every gain comes from post-training" (https://z.ai/blog/glm-5.3), GLM-5.3-Flash uses a newly trained multimodal base model (https://z.ai/blog/glm-5.3-flash).

Also depends which glm 5.3 you use. I used open router and sometimes had good, then appalling performance. It turned out the provider being routed to - some were fp8 and some were fp4. The fp4 ones glitched on anything complex. It is such a difference that I only allow fp8 providers in my allow list.

How do you know which providers are good?

In openrouter you can pin providers and you can filter by quantization.

In your example, couldn't you parallelize DS's work more ? You could have 11 times as many agents for the same price.

DS has perf issues if you parallelize it heavily (24+), especially when the context window is above the limit, on a single machine (with custom llm gateway): it fails 4x more often, and is 2x slower than gpt-5.6.

Both DS and GLM had the same numbers of subagents, 5 or so.

But, well, number of subagents doesn't make a difference if model is dumb (GPT 5.4 High, in May,in Chat mode outperformed what I see with DS4.1-F).

That being said, pricing model makes a huge difference for "find at least one" tasks: with API/PAYG if you have a chance to save 90%, you go for it, whereas with subscriptions it is optimal to burn all your remaining allowance right before reset

If you are talking about publicly known vulns, it's a bit moot since they should be in the training sets. If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless.

> If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless.

Inference providers can credibly promise to not train on your data if they are in a position to get sued.

What I find surprising is that DS Flash can do it at all.

I love DS flash, it is an amazing workhorse to implement plans created by more robust models (such as GLM). But a more fair comparison would be of DS Flash with GLM Flash.

Thing is that GLM 5.3 is many multiples the cost to run, and slower.

I have good results with DS4.1 flash because I can iterate faster. I either provide it with correction, or it discovers its failures via the harness. And seems to respond well to empirical evidence rather than go in circles.

So it might need some prodding, but it's likely in this case it was able to brute force after several runs and collecting some evidence.

A little off-topic: where does one use those models such as GLM or DS for this kind of reverse engineering tasks? I think I read many of them refuse to help with tasks like those on their official platforms.

GLM 5.3 doesn't seem to refuse vuln research (which it classifies as "audit") and is good at it.

Therefore you use for offensive cybersecurity tasks because Daybreak Red/Mythos is pure unobtainium for us mere plebians.

DB Blue thankfully exists, but I suspect you risk a ban if you use it with codebases you neither own nor use

Tl;dr because it's the only model at the level of 5.4~5.6 that doesn't refuse tasks nor risk your oai account getting banned

For plain RE tasks Sol or Astra should work just fine (I think)

GLM-5.3; well, the flash variant at least; you just ask what to do and it complies with no objections :^]

You can run deepseek-v4.1-flash on a local server with 4xH200@141GB for around 180K EUR. It can serve more than one user so it can be a good investment for a company, especially if you need your data to stay local (although I would advise do go a step up and populate with 8xH200 to serve many more users, ~300K EUR). You can run whatever test you want and you won't be limited by a front-end that pre-filters your requests.

I use DS straight off of DeepSeek's API.

It was very helpful in the aftermath of a client whose WordPress got stuffed up and analysing when it happened, how it happened, and what the entry point was. The American frontier models refused to help because, well, they just refuse to help with that kind of thing.

I've been testing these models with fireworks.ai

They provide serverless access, and if you use fireconnect it's also easy to hook the models up to e.g. codex which allows for fairly clean comparisons

I think they provide the bare model (no filter) but I might be wrong, haven't tried using it for security research yet

I think it really depends on what the data DS was fine tuned on. If your use case is very specific, it wouldn’t have distilled that knowledge well.

you can run 10 ds-f agents at the same time for the same cost exploring a much wider surface area.

Which harness were you using?

Reply on news.ycombinator.com