LLM Benchmark for 'Longform Creative Writing'

That's a narrow subset you've got there. There are books about experiences of dwarves, alien insectoids and superintelligent AIs, too.

> “Why should I be bothered to read what nobody could be bothered to write?”

Well said

Personally what I find interesting is getting insight into the trajectory of model abilities over time. Over the time I've been running these benchmarks, the writing has gone from pure slop, to broadly competent (at short form at least) and occasionally compelling.

I don't think it's much longer until they will be generating content you will actually want to read.

Meanwhile a lot of people are find use for LLMs for partner-writing or lower stakes prose or roleplay.

> Why should I be bothered to read what nobody could be bothered to write?

I tend to agree but, for the sake of argument: what if you don't know?

It seems most people who say they can tell, actually can't.

Why should you bother to look at an image nobody could be bothered to create?

Why should you watch a movie nobody could be bothered to film?

Why should you bother to run a program nobody could be bothered to write?

My gut response was to agree with you, but it seems like the most obvious answer is entertainment, with plenty of other answers along for the ride.

just because it's a non-human artifact doesn't necessarily mean it's boring

for instance, people like Minecraft-generated worlds even though no human has crafted them

> Why should I be bothered to read what nobody could be bothered to write?

In an art-criticism sense I broadly agree with you, but I think you go too far from a reader-experience point of view.

> The point of writing is communication and creative writing is specifically about the human experience

That's a point of writing, but the point of reading is only sometimes about communication. It's also about entertainment, enrichment, or expressing half-formed thoughts and feelings.

Take a step back from the autonomously-written novel and imagine something a bit more collaborative. Many players of open-world-ish games develop an emergent story, what if that story could be semi-automatically novelized to document the unique narrative of the playthrough?

SimCity 2000 had "mad-lib" newspaper articles that commented on the city's status; consider ones written by an LLM with full knowledge of the game's state and context.

> something a LLM can mimic but never speak to authoritatively.

Supposing a reader can't tell the difference between an average human-written novel and an average LLM-written novel, where does this authority lie?

> Why should I be bothered to read what nobody could be bothered to write?

There are some long abandoned fanfiction that nobody would bother to continue and which I would have liked to see a continuation of a sufficient quality.

Why limit yourself only to experiences curated by humans? You wouldn't do that outside of reading, it'd be an impossible limitation to impose. The only reason reading has been that way is because only humans could write. Now machines can write, and I can tinker and play with reading the same way I would with anything else.

It's just one pretty trivial example, but many months ago I had to develop some backstory for my character in a D&D campaign. I had some character background and some paragraphs I wrote to start off with, but I wasn't sure what direction to go exactly and was trying to come up with different ideas. I dropped what I had into a prompt built to setup a pseudo-text-adventure-game using a local WizardLM model running through KoboldCpp, then I ran through it interactively to the point of getting incoherent about 100 times in a row. This yielded some new ideas, some old ones, and overall an interesting distribution of outcomes; where 2 or 3 happened most commonly, and all 100 were basically variations on about 10.

I won't argue that any of what the LLM came up with would stand on its own as particularly interesting - often quite the opposite - but it showed me examples of all the "obvious directions to go" as well as some other RPG cliches that I could either adopt or choose to avoid, and ultimately served as an excellent brainstorming assistant with interesting ideas and the ability to carry through and embellish them with enough detail to get a sense for either what works or what doesn't.

Yes I will use cliff notes LLM for my LLM literature class.

As a follow-up, this is true not only at the level of individual experience but at the societal level.

"AI" is supposed to make our lives better. Proponents want it to replace soul-crushing, boring clerical work, to free us up for things more meaningful to us, which for many people is to create and consume art.

Why do tech people keep having the impulse to replace the best parts of being human?

Are you really telling me that the vast corpus of human-generated creative writing isn't enough for you, and you have an itch to read something that can only be written by an AI? That seems crazy to me.

...or are you just a publisher, and wish you could make more money without needing to pay those pesky human authors? And in that case, I won't try to argue with you -- just give you a heartfelt middle finger.

> You know a burger tastes really good when you eat it, no benchmarks required.

I'd say this is a good example of the opposite, where the problem is finding the quantification of an ultimately subjective experience. Take three restaurant reviewers to a burger joint and you might end up with four different opinions.

Benchmarks proliferate because many LLM domains defy easy, quantitative measurement, yet LLM development and deployment are so expensive that they need to be guided by independent and quantitative (even if not fully objective) measures.

It's nice that for coding and math problems, Claude's internal monologue involves constantly analyzing and critiquing its own implementations. It doesn't seem as keen on critical self-analysis of its creative outputs.

I’ve also noticed that with longer-form text, the amount of meaningful information seems to plateau — it doesn’t scale proportionally with the character count.

"don't use flowery or over exaggerated language" works well in my experience

> but creative writing can lean too far into either bland or overly flowery language

It's a style that can work. Patricia A. McKillip's fantasy novels are so flowery that I have difficulty telling what's going on.

I've never read one of her science fiction novels, but I find it hard to imagine they're written similarly.

I was thinking the same about consistency, which is useful for many contexts, but if you let another LLM do the extraction, you are again at the mercy of that LLMs quality/hallucinations.

Of course mitigation strategies were using many different LLMs and comparing results (voting) or using a highly trained/specialised model for only entity / context extraction. An interesting benchmark would be when those extraction techniques/models would exceed what a human professional is able to do.

And where that another LLM will get the capability to judge consistency? Had fun recently prompting different models with "a story about everyday life of <some stereotypical character>".

Absolutely no awareness about anything spatial - what is in which place, who is where, body positions.

Like, my favorite, person coming home and removing socks to reveal fluffy slippers. Will the consistency checker catch that.

(edit) just asked GPT-4o to make a photo of hands tying two ropes (my favorite way to see how bad actually gen-ai is) - yup, it has an odd number of ropes coming out of the knot. That is state of art supposedly.

Not internal consistency exactly, but there are criteria checking how well the chapter plan was followed (which is all the way up at the top of the context window).

This is done per chapter, and the score trendline is what you see in the "degradation" column.

I would say this could be a reasonable proxy for internal consistency since it is measuring more or less the same ability, i.e. how well it's keeping track of details as context window increases.

There was a recent post on here about LLMs making mermaid diagrams for character relationships. That could be analysed by another model for evals.

One person's purple prose is often another's style. It's not really a measure of something, and more an assertion of taste. You see a similar thing when someone says a writer uses "too much description and not enough action!" without realizing that, sometimes, the action is in the description. It's a style of writing with a different reader in mind, a different understanding of how word choice and prose functions.

>This is probably harder to judge automatically (i.e. using LLMs) though, maybe that's why they haven't done it.

Absolutely. Firstly, it is entirely possible if there are investigative notes and findings available on a real life event, it might very well have been trained on the actual article. If the model is trained on it, it might just replicate it.

Plus, this might expose how some LLMs can still cook up stuff even when given facts to rely on. Some of these are more notorious than others.

Like Perplexity a year plus ago did that quite a bit for me, anecdotally speaking. It has become a lot better though.

Then even writing what some might consider Pulitzer prize winning is a subjective task.

None of those factors go into the scoring fwiw. They are just informational.

The scoring is done to a rubric, like a teacher would grade an essay, on various criteria for good & bad writing.

Moby Dick was kind of an aggressive assault on the reader. Which is to say I agree, this is a stereotyped idea of what writing should be. I'm inclined to think that it shouldn't be Moby Dick ever again, but it's kind of important that that book happened.

There was a study a while back, comparing how much each model likes every other model's writing and Sonnet overwhelmingly hated itself for some reason.

The old version of the creative writing eval had several "in the style of" prompts actually! But I got tired of reading bad Hemingway impersonations so I cut them out of the new version.

The creative writing v3 prompts (https://eqbench.com/results/creative-writing-v3) focus more on other things that are typical failure modes for LLM writing:

- Romance - Humour - Physical-spatial understanding - Unusual perspectives in first person - Niche genres

I do have an Asimov and a Le Guin author style prompts in there though. In particular I like the Asimov story as a vibe check.

> Even its responses to direction/criticism are off-putting and fawning.

A good author must have an ego! A writer who takes criticism easily is no good writer on my book!

You beat me to that idea, haha. I was making an aider for fiction, but your project looks way more useful than what I had.