Show HN: Evaluating LLMs on creative writing via reader usage, not benchmarks | Hacker News Reader

Show HN: Evaluating LLMs on creative writing via reader usage, not benchmarks

narrator.sh

36 pointsjauws12 comments

Hey HN! I'd love to get some people to mess around with a little side project I built to teach myself DSPy! I've been a big fan of reading fiction + webnovels for a while now, and have always been curious about two things: how can LLMs iteratively learn to write better based on reader feedback, and which LLMs are actually best at creative writing (research benchmarks are cool, but don't necessarily translate to real-world usage).

That's exactly why I built narrator.sh! The platform takes in a user input for a novel idea, then generates serialized fiction chapter-by-chapter by using DSPy to optimize the writing based on real reader feedback. I'm using CoT and parallel modules to break down the writing task, refine modules + LLM-as-a-judge for reward functions, and the SIMBA optimizer to recompile user ratings from previous chapters to improve subsequent ones.

Instead of synthetic benchmarks, I track real reader metrics: time spent reading, ratings, bookmarks, comments, and return visits. This creates a leaderboard of which models actually write engaging fiction that people want to finish.

Right now the closest evals for creative writing LLMs come from the author perspective (OpenRouter's usage data for tools like Novelcrafter). But ultimately readers decide what's good, not authors.

You can try it at https://narrator.sh. Here's the current leaderboard: https://narrator.sh/llm-leaderboard (it's a bit bare right now b/c there's not that many users haha)

(Fair warning: there's some adult content since I posted on Reddit for beta testers and people got creative with prompts. I'm working on diversifying the content!)

Open articleView on HN

Btw, creative writing is something where good sampler settings uniquely improve your experience a lot. That's why the coomer/ERP crowd is usually the first to implement a new sampler technique.

You should explore high temperature (far above 2) sampling with good truncations like min_p, top n sigma, TFS, mirostat, typicality sampling, etc. Basically anything that isn't top_p/top_k. This is the path to highly diverse outputs.

This is an amazing suggestion! Will definitely try to figure out a way to incorporate this into the leaderboard without making it a constant each time. I'm currently using OpenRouter's default parameters which is totally a brainfart on my part.

I run a site that does something similar, but on a more granular level (prompts at the page level rather than the chapter)

I think right now we're at the point where novelcrafter is an excellent proxy for the best models for readers, because LLMs are still mostly losing engagement due to technical errors as opposed to subjective ones:

That's repetition problems, moralizing/soft-censorship, grammatical quirks, missing instructions, forgetting major plot points, etc.

Those kinds of errors are so obvious you can almost rank these models with an N=1 vibe test, and they limit how much people will consume unless you're scratching certain itches like NSFW

-

However I do think with enough post-training you can beat that level of problems and move to a stage where the writing is technically sound (and that's what I've spent most of the last year working on).

From there you get to more challenging problems that require much more feedback along some level of specialization per user (like what Midjourney does during onboarding to build up a style profile). Once you're not making technical mistakes, you now have to codify the ethereal concept of "user taste", and that will be a really interesting challenge for LLMs.

Thanks for the comment! Do you mind linking the site - would love to check it out! That's a very fair point about the technical error aspect. Though with all the confounding variables (author skill differences, model selection based on price/speed, etc.) I'd say it's probably the most mature signal we have right now, but still far from ideal.

Really interested in what you've been working on for the past year! Are you doing custom fine-tuning or more on the prompting/post-processing side? Also I definitely need to check out the Midjourney onboarding, it sounds super interesting for inspo regarding your point about personalization + taste!

My 2nd most recent submission has a link to it

Most of it has been fine-tuning (SFT/DPO/GRPO), but also a lot of prompting and adding steps between the user's prompt and the output

Nice approach! Reader engagement beats synthetic benchmarks any day. Bookmarked to try later - curious which models actually hook readers vs just score well on tests.

Thanks Johnny! I totally agree with you, really appreciate you for checking out my project!

Really cool project, and the site looks clean too! Have you thought about tracking where reader drops off in a chapter? Could be great signal for narrative flow.

The plagiarism tumbler turns and turns.

> currently in early access. just a fun side project :)

This causes me cognitive dissonance.

Really like the shift from synthetic benchmarks to actual reader engagement — feels way more aligned with what “good writing” actually means. Curious if you’ve noticed certain models consistently improving more with feedback than others.

Thanks! Anecdotally, I'd tend to say that Claude 3.7 tends to improve the most, but it seems like (via the leaderboard), some people really prefer Grok-3 lol.

Reply on news.ycombinator.com