Hacker News Reader

Comment by Systemerror7A69 | Hacker News Reader

Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work?

I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece.

I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentence prompt.

So, it seems to me this "oneshot from a simple prompt" eval is fairly meaningless when it comes to model evaluation itself, as it is in no way representative of real world application.

This would be more something for "vibe coders", people with little to no programming background wanting a website?

View on HN

I checked out the development benchmarks for agents, and was suprized to find that they're not really representative of many of my own day-to-day development workflows.

It appears that they're mostly testing the ability to make business tasks autonomous, with ~20% associated to development tasks (ssh here, install this, etc.), but not actual programming.

That's probably because the actual coding benchmarks were saturated several years ago.

Which is why you should perform your own benchmarks against your own software stack.

Agree. To this I would add, many things are saturated even for smaller models, which tend to be much cheaper and faster.

On many of my tests, there was no difference in the result between the smaller and bigger model, but there was a big difference in speed and price.

Seriously, I can't stand these tests of 1000 input tokens generating 1 million output tokens. You can't really convey anything interesting in 100 words, so like almost who cares what it produces? No one is providing value if the extent of their contribution is so small.

Much more interesting is providing a million tokens of meaningful input and getting 1000 tokens out (high level critique of a detailed design doc, finding a subtle bug in a big codebase, etc).

I think both can be interesting, but surely the 100 words will lead to a lot of variability in the exact shape of the result because so much of the spec will be left up to the LLM. However, it is still interesting to judge the quality of the result and how self-consistent it is. Not by looking at the actual webpage, but by looking at the code and whether it is nicely organized, makes proper use of abstractions, and in general doesn't feel like spaghetti.

Great - can you show us some examples of these? Every time this topic comes up here it’s full of people saying “you’re using it wrong” and apparently it should just be intuitive. Yet all the material is like this.

As someone who's done web development for 20+ years, I find the model personalities pretty dang fascinating, especially how they develop (and evolve) design sensibilities.

I'm interested to know how the classic AI "purple preference" emerged (organically?) and if the beige wave came out of specific training efforts to combat it?

To your point on development work (the code itself), I was talking to some friends on the Google Chrome team about any research understanding the model's preferences around framework ergonomics and abilities to properly implement core web standards for given tasks. I think that would be super fascinating.

I'd assume it comes from Tailwind boilerplate/template sites as it seemed almost all of them were purple at the time

Definitely. It was already a trend before LLMs exploded. And the whole "all websites look the same" has been a thing since at least Bootstrap times.

Purple is just colors #101. ;)

It's impossible to NOT get the beige/orange/cream/paper combo.

You basically have to explicitly say not to use those colors/themes.

Beige+Orange 100% came from Anthropic intentionally training this, it's their own theme. I don't remember what their first release was that defaulted to this - it may have been Sonnet/Opus 4. Before that, not one other model used it, nor was it a particular popular theme. For light themes, white and off-white had been ultra-dominant. Anthropic themselves were the outlier with beige, which is also why the change to beige stood out so much.

I'm repeatedly noticing that people working at big ai and tech companies are surprisingly not that... good... at using ai? It's like theyre doing a plausible thing to get something done and calling it a day

Can you share some posts of good examples of prompts and comparisons?

The eval world is split into:

1. Long form task based examinations like this that test the ability of the model+harness to remain on task, tool calling, overall effectiveness and taste.

2. More direct 1:1 and qualitative comparisons that you might get with a tool like https://evvl.ai/ - which also uses OpenRouter and does similar one off model comparisons (or lets you use it as a MCP from your dev env to be like: "take the prompt from this loop and try it against these other models")

I've been using code review as my benchmark. Launched a massive parallel Fable/max code review on my lone lisp codebase and recorded all the results and transcripts. Switched to OpenAI and am now running the exact same code review with Sol/max.

It's still a work in progress but preliminary results reveal that Sol is able to reproduce 70%-90% of Fable's performance. This is a very meaningful result for me because code review is what I use AI for.

That would be a very valuable benchmark across different models.

You touch a point I quickly skimmed in another comment.

Yes, the most valuable benchmarks and evaluations you can write are those that resemble your work.

The evaluations are extremely hard to write and test.

And yes, virtually all benchmarks are E2E one shots, they do not reflect multi turn processes or how most people interact with LLMs.

Which is why every Opus after 4.6 looks better on benchmarks, but is hard to work with interactively.

> Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work?

As a non-designer trying to build something solo, I think value's in the higher-tier models being able to brainstorm and infer a variety of design directions, as well as dissect the "this looks off" comments that would frustrate human designers.

Some of the frontend design skills try to bridge the gap, but the better models perform far better as is, and often better without one of those frontend design skills trying to ram their own workflows in.

Yes, I would be very interested to see follow up prompts that adjusted visual elements as if feedback from the client, then some prompts to change the menu and pricing. Id like to see how well the AI's can maintain their own work.

I've been super impressed with one shot AI images and designs in the past, but have never been able to adjust a design without things going off the rails.

> Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work?

No, you are not wrong. If you now crack the "What is?" in a generalizable way, there is very good money in that.

Reply on news.ycombinator.com