Comment by stusmall | Hacker News Reader

Comment by stusmall | Hacker News Reader

Parent Full thread

I'm generally extremely skeptical about a lot of the model hype that show up in comments. Except when there is an extreme mismatch the performance, quirks and quality of these things are difficult to nail down. You wouldn't know that from the comment section of every single release.

I think some of these are excited, eager users always ready to hype up the new thing. The same crowd that previously would constantly push for a rewrite from angular->react->svelt->god knows what. Instead now it is on a 6 week cycle and about models/harnesses.

I think most of it is bot driven spam by the various labs. It's hard not to notice 3 month old accounts with very strong opinions about various frontier labs and little else.

Ultimately, I think some of it is legitimate shifts in who's in lead and what is the best. You gotta dig through a lot of crap to get to that, and I don't really know how to to.

Ultimately I'm saying is that I always applied a fair amount of skepticism about what I see in comment sections but these day it is extreme amounts.

> I think most of it is bot driven spam by the various labs. It's hard not to notice 3 month old accounts with very strong opinions about various frontier labs and little else.

I've assumed the same as well.

I also assume that many of the companies developing these models engage in benchmaxxing.

At my company we've developed our own internal benchmarks for evaluating LLM models as they become available. The benchmarks are tailored to our particular use cases but the utility and knowledge our benchmarks assess is still fairly universally applicable. I see wide differences between what our internal benchmarks report and what the major benchmarks do.

There was a whole lot of fanfare about how amazing GLM 5.2 was when it was released, but it was pure rubbish on our internal benchmark -- far behind OpenAI, Anthropic, Gemini, DeepSeek, etc. I don't know how to reconcile the fact that GLM 5.2 performed very well on some of the major public benchmarks, but consistently performs so poorly on ours. OpenAI models tend to dominate our internal benchmarks.

It actually is showing in public benchmark if you know how to look for it. For example, in Terminal-Bench 2.1, GLM 5.2 received 78%, while GPT 5.6 Sol received 88%.

Then Terminal-Bench 3.0 came out (where the questions are new), and GPT 5.6 Sol received 34.6%, while GLM 5.2 dropped to a whopping 4.6%.

> OpenAI models tend to dominate our internal benchmarks.

That's odd, since Fable seems to be the leader for the industry. Not cost-effective, but if Anthropic models get dominated by OpenAI in your internal benchmarks, this calls their validity into question. Separately, see the jagged frontier effect. [1]

I've been using GLM 5.2 at my day job (mostly Rust backend work ATM). Nothing that blows away the models from OpenAI and Anthropic, but solidly good enough to get it done. A lot of people have experienced this and the fact that an open weights model can do so is where most of the excitement comes from. Optimizing for benchmarks can only get you so far, and people are quick to criticize models that fall into it (like DeepSeek Pro V4 recently).

[1] https://mitsloan.mit.edu/ideas-made-to-matter/working-defini...

This resonates. You need to tailor evaluation to your own use cases in order to know if you should adopt the next thing or not.

I think most of it is bot driven spam by the various labs. It's hard not to notice 3 month old accounts with very strong opinions about various frontier labs and little else.

Really no different than when there was suddenly online personas everywhere hyping up TSLA out of the blue. You can see the same thing going on with the BoringCompany subreddit. Crazy that the botnet master isn't able to convince us that Grok is also the best model. I don't think buying twitter was an accident it was probably just literally covering up the evidence.

https://www.rhsmith.umd.edu/research/twitter-bots-boost-tesl...

IMO this is getting hyped because the 27b version runs on a decent gaming GPU. This is NOT a thread for their largest model, this model will run on a mid-high end gaming PC, which you probably have in your household. Mine is 6 years old and it runs quite well.

I strongly suspect that many of these accounts you think might be bots from the labs are just people who only have one interest.

One thing I have noted a lot more of is that there is comparative fanboying going on. Like "this model has done badly, my favoured competition has a model out soon that will beat this in every way" — comparing a released product to unverifiable hopey claims about an unreleased product.

It's tempting to assume that is bot stuff, but if you've been around any other "hot" technical hobby online (cameras, phones, 3d printers, whatever) you will know it's not. It's just fans aligning into teams, some of them laconic and amusing, some of them overkeen and toxic.

Reply on news.ycombinator.com