An Introduction to AI Story Generation
thegradient.pub
thegradient.pub
Symbolic story generation systems were capable of generating reasonably long and coherent stories. These systems derived much of their power from well-formed knowledge bases. But these knowledge bases had to be structured by hand, which limited what the systems could generate. When we shifted to neural networks, we gained the power of neural networks to acquire and make use of knowledge from corpora. Suddenly, we could build story generation systems that could generate a larger space of stories about a greater range of topics. But we also set aside a lot of what was known about the psychology of readers and the ability to reason over rich knowledge structures to achieve story coherence. Even increasing the size of neural language models has only delayed the inevitability of coherence collapse in stories generated by neural networks.
A primer such as this one makes it easier to remember the paths that were trodden previously in case we find opportunities to avoid throwing the baby out with the bath water. This is not to say that machine learning or neural network based approaches should not be pursued. If there was a step backward it was because doing so gave us a powerful new tool with the potential to take us further ahead. The exciting thing about working on automated story generation is that we genuinely don’t know the best path forward. There is a lot of room for new ideas.
My personal view is more critical than the author's. There was an approach that worked reasonably well but needed more work to improve it. It was abandoned for a flashy new approach that doesn't work and doesn't seem like it could be made to work ("Even increasing the size of neural language models has only delayed the inevitability of coherence collapse in stories generated by neural networks"). Why was the working approach abandoned? The answer seems to do with a certain tendency in academia to blindly follow new trends with the only criterion being how many published papers can come out of it. That is no way to make progress. That is the way to waste time.
Besides, this is not true:
>> There is a lot of room for new ideas.
Once your field has been taken over by neural networks, there is no room for new ideas. You are destined to tweak hyperparameters for the rest of your academic career.
To clarify I didn't mean that neural network approaches don't work. I meant that they don't work _for story generation_ certainly not if the goal is to generate coherent stories. In the article, the best examples of stories generated from neural network approaches (basically, nn-trained language models + x, where x is various clever tricks that try to patch up the gaping holes in neural nets' inability to generate a coherent story) look decidedly worse than the stories generated by symbolic approaches listed in earlier sections.
In my mind, the author of the article is being polite and trying to present the latest work in his field in a positive manner, possibly because of the natural tendency of researchers to not take a huge dump on each other's work. But some of the expressions he uses, like "in case we find opportunities to avoid throwing the baby out with the bath water" make it clear that he doesn't see the takeover of his field by neural networks as a step forward.
Btw, "cynical"? Should I take that as a personal comment? It sounds a little insulting.
I'm always impressed by what it comes up with, though not all stories are 100% coherent.
Curious-- are you "invoking" the model for every user or caching some of the results?
I've been obsessed w/ using GPT3 for "personas" in conversations (for the agent to take on a character or follow instructions), but didn't seem feasible to make an API call (esp to Davinci) for every user
It caches every result and only invokes OpenAPI upon clicking the "Generate More" button. The response is then saved for all future users of the site.
So far everything on the page has been generated for under $10. But I can see how it would become prohibitively expensive if generating unique text for each user.
There may be some way to carefully design prompts so that models learn how to both summarize all past events, which could then used as context for subsequent events. And also, to train them how produce coherent story summaries and then use those as a nucleus to develop properly structured stories - but we’ll need to investigate :)
Here are a couple stories that I would consider finished:
1. https://toldby.ai/mfuVU3z1uUa
2. https://toldby.ai/Jjl1fpW9xUO
3. https://toldby.ai/CTq_fko8CUS
I'm considering adding an option to "End story here." And then have GPT-3 generate a title and prevent further branches.
Of course, there's a lot of problems with this approach also, but it does help specifically with consistency across larger generated works.
Since models are becoming centralized, it’s a bigger deal than it seems. It’ll be interesting to see whether the AI community settles on one specific list of “don’t”’s, or if each company will have their own variant.