Hacker News Reader

Comment by miltonlost | Hacker News Reader

These Benchmarks are asinine to determine the quality of "creative writing", all chosen by engineers without a single artistic bone.

Length

Slop Score

Repetition Metric

Degradation

Moby-Dick has chapters as short as a page and as long as 20. According to this benchmark, the book would score lower because of the average length of its chapters.

These aren't benchmarks of "quality". A chapter's length is not indicative of a work's quality. That measurement, on its own, is enough to discredit the rest of the benchmark. So so so so so so so misguided.

View on HN

Moby Dick was kind of an aggressive assault on the reader. Which is to say I agree, this is a stereotyped idea of what writing should be. I'm inclined to think that it shouldn't be Moby Dick ever again, but it's kind of important that that book happened.

None of those factors go into the scoring fwiw. They are just informational.

The scoring is done to a rubric, like a teacher would grade an essay, on various criteria for good & bad writing.

None of those factors go into the scoring? What's the point of showing them then? What information are these factors providing if they aren't being used in the rubric? Without knowing the rubric (which isn't provided), these scores are baseless to me, and I don't know how to use them. Blackbox numbers are not useful.

All the judge outputs (including rubric) and model outputs are in the samples reports.

Sorry you don't like the displayed metrics. I find them very useful / revealing of the things I'm trying to measure with this benchmark.

These are important things to look at because they're where LLMs typically have trouble. People complain that their stories are too short, that they use stock phrases and purple prose, that they start to repeat themselves at a certain point, and that quality tends to fall off. Doing well in these problem areas doesn't guarantee good writing, but it does allow for it.

Reply on news.ycombinator.com