The computers are getting better at writing
newyorker.com
newyorker.com
I think the next step (say, being able to produce short stories with actual literary value) will be substantially more difficult, mostly because it's substantially more difficult for human writers too. It takes learned abstract knowledge and deliberate intention to write something that's coherent and engaging and longer than a few hundred words, and that skill is separate from the intuitive prose generation ability that's essentially unconscious for practiced writers and comes easily to many avid readers (i.e. trained on a large corpus) who take up writing.
It will be interesting to see how producers of actual substantive content incorporate these tools into their processes, if at all. I guess I can imagine using it to generate descriptions, for instance, but how does it do with dialogue? Probably not very well.
Which leads to a simple improvement: teach the model when to stop, what sections to remove. There’s already great ML-based summary engines, and systems able to answer questions from descriptions.
GPT-3 reads like the unfiltered garbage that comes out of my head when I'm trying to get ideas out. The primary difference is that I have empathy for the audience so I go through the effort to reread, clean it up, clear it up then cut it down.
But that secondary editing process has nothing to do with the subject matter or initial content generation process. It was trained through years of *not* doing it and having my audience ask lots of questions and clearly misunderstand what I was trying to say (let's call it reinforcement learning). As a result I now have an intuition about which concepts need explanatory analogies, based on my understanding of what the readers know, and which are almost certainly going to be understood outright. I think this gets at the root of the difference.
Basically when I write, I'm not trying to generate text, I'm trying to generate training data that will amend foreign models with a high fidelity representation of a subset of mine.
This feels GAN-like but different. It's like you need to have two separate models, the 'expert' and the 'n00b' (or ten n00bs trained on different data), ask both of them the same thing, and one-shot retrain the n00b from the answers provided by the 'expert' until the n00b is able to generate content that the expert finds to be sufficiently similar.
Well said! Replying so I can find this thread again, please ignore.
People who are afraid of the current generation of ML writing are absolutely dogshit writers with terrible taste in writing. I keep forgetting who said this, but theres a saying to never start writing by describing the weather. Lots of folks miss the point to this. You could modernize it by saying, "Dont start talking about your childhood when writing a recipe." Why? Because no one fucking cares. They want the recipe. Not a description of your childhood or what the weather was like. Take the time to notice articles and "journalist writing". If it starts with the weather, notice how incredibly long it takes for the author to get to the point or describe anything functionally meaningful. You'll notice these articles are extremely long in word count and extremely low in actual information/content. These are the people afraid because yes, a machine can keyword drop and fluff better than them.
Is it possible that there will be a future AI/ML to write with purpose? Maybe. But realistic idea connection and logical flow is still a huge hurdle for AI/ML. Summary engines seem good on the surface, but again, if you actually read them for a real world reason, not for "look how good ML is", you'll notice they're as meaningful as a politician running for office. Lots of keywords, lots of nothing really said or done... just like anyone with a liberal arts degree.
Absolutely. It helps make clear that the function of words isn't always what we think. It's the sort of window into the machinery of the mind that helps us reverse-engineer what's really going on.
It reminds me of the Oliver Sacks essay "The President's Speech", which Wikipedia describes as being "about a ward of aphasiacs [people who have lost the ability to understand words] and agnosiacs [here, people who understand only the words, as if reading a transcript] listening to a speech given by an unnamed actor-president, 'the old charmer', presumably Ronald Reagan. Many in the first group laughed at the speech, despite their inability to follow the words, and Sacks claims their laughter to be at the president's facial expressions and tone, which they find 'not genuine'. One woman in the latter group criticizes the structure of the president's sentences, stating that he 'does not speak good prose'."
Glibness has a purpose, but it's not really about conveying information. It's for something else.
agree. I'd say it's impossible to solve. Where would we even place our yardsticks:
Has society been able to get any better (than we used to) at assessing and categorizing what "constitutes" great writing? If so in what relation could that be measured to our past performance especially that many no longer read today due to our loss of attention span? (most won't even read this comment because it's way too long).
People are terrible at spotting some of the best writing when it happens. History is full of writers who's work could only be understood and appreciated after their death.[1] Death makes their work a finite commodity. There is no more chance asking the author "what they meant". Others get shunned because they were too radical, (or too ahead of their times).
Great writers as in "a radical new style" or as "for their radical ideas" get only credited after decades (if not centuries) post mortem. When it's too upsetting (because it's true) we need the distance of history and later generations to appreciate it[2].
Even "what" we comprehend and then appreciate changes over time, and within the same person with every time we read it[3].
[1] 6 people centuries ahead of their time https://www.oxford-royale.com/articles/6-people-centuries-ah... <- there are hundreds of wrtiers like this (it is just the top result google gave me). How would an AI spot it when all AI was created "in our image". And if it did spot it how would the AI be able to convince us that this is worth a closer look instead that we need to destroy the AI?
[2] The Last Messiah by Peter Wessel Zapffe https://philosophynow.org/issues/45/The_Last_Messiah <- say something mediocre to ensure you get published and ranked in "best new books". But saying something profound can potentially get you ignored (at best) and killed (at worst). This ties back to[1]
[3] James Joyce Dubliners https://archive.org/details/dubliners00joycrich / https://en.wikipedia.org/wiki/Dubliners <- there are hundreds of examples like Dubliners but they are all different books depending on who reads them and when. the retrospect and integrating what we read (and then re-read over and over at various stages of our lives) into our own reality is what is makes it great. yet even if some minority of people does practice this, it's still highly subjective. how would an AI provide us with value (e.g. if it can the value it creates would only benefit itself but never others)
__
I'm not saying AI is incapable of better writing than AI is now, but it would only remain mediocre and never outperform humans. Also the writing can't stand or be judged by itself. It can only be valued and appreciated through human mind. Art (and many books are art) has the same problematic relationship with AI. John Berger's "Ways of seeing" talks about this (but so does a lot of philosophy) https://www.youtube.com/watch?v=0pDE4VX_9Kk
I admit I was going to skip it, but I saw a reference to a James Joyce's book and reconsidered :)
If the goal is to increase the SNR, GPT-3 is probably not going to help.
> I think the next step (say, being able to produce short stories with actual literary value) will be substantially more difficult, mostly because it's substantially more difficult for human writers too.
Totally agreed. Both my co-founder and I are professionally published short story authors, and we would be the first to say that using something like Sudowrite is no guarantee you will land a story in the New Yorker. Similarly, using Cinema4D won't propel you to the same quality work as Beeple. These are tools. It's up to you as the author to use them in a way that enhances your craft.
My goal is to figure out how we mindfully incorporate AI into a writer's workflow. If you look at other fields like visual arts, they have had a wealth of interactive tools starting with photoshop. What is the photoshop for writing? (hint: I don't believe it exists yet, and I want to build it)
In terms of the details, you'd be surprised at how well GPT-3 handles descriptions and dialogue. It's actually scarily good at dialogue [2] [3].
But then again, aren’t we all doing the same, regurgitating bits and pieces that we heard or read before and liked?
Don’t we all sometimes, when busy with our own thoughts and worries, occasionally spout out oddly appropriate responses to what our spouses are speaking, without actually listening to and processing the conversation, maybe because we heard it all so many times we can readily predict what they are going to say? And then we agree that we need to go take the alligator from the repair shop.
However, if anything, GPT-3 is mostly showing how little meaning there is, but in how many words.
Never mind me, I’ll pour myself another one.
Uh.. no, not me. That sounds very strange to me, and sad. I always listen to everything she says. And expect the same back. Otherwise it wouldn't be a good relationship, I think. (Maybe I'm super-naïve!..or something.)
No. Sometimes I can predict what a friend will say but I won't actually say it. The more I think about it, the more I realise that AI at the moment is the unlistening (but hearing) spouse. And just like it seems we can engage with such a person (though can only rarely), so too can the AI trick us ('simulate') that behaviour.
We can do that. We can also not do that. Language models can only do that.
My comment is stripped of nuance (language models don't do that exactly) but it should point at the difference: humans can be as dumb as bricks, or as smart as humans. Language models don't even come close to our brighter moments. They don't even come close to the brighter moments of our pets, and our pets can't generate human language.
You have kids? Think of whom you'd like to spend the next decade or two around: a five year old human or GPT-3? Which one would you find more interesting, original, curious, do you think?
I think people who are the most excited about GPT-3 are the ones who are used to even dumber, more boring interactions with computers than the repetitive and unoriginal text generation of language models. Or perhaps it's humans who are used to boring interactions with other humans. Technology has done some weird things to us of late I fear.
Kids and language models are very different kinds of "interesting". I like to be around my child, because I love her and it's good for the soul. If I only have one time slot and a choice who to spend it with, GPT-n can go pound sand, I'll be with my kid.
A language model can certainly be fascinating in other ways. It brings me into a philosophical mood and spurs some questions about the humanity and quality of human knowledge, and efficiency of passing ideas around, but those don't imply answers I'd like to have, and tend to eat my soul away.
> However, if anything, GPT-3 is mostly showing how little meaning there is, but in how many words.
We imbue words and symbols with meanings, and we strive to give our stories meanings that transcend the prose on paper. Yes, it's true that GPT-3 can spout non-sequitars. But overall, it does stay on topic quite well, as compared to previous iterations like GPT-2.
But sometimes, you WANT that, you want to be provoked to get unstuck on writing that tough scene or character. I've found that using a system like Sudowrite is not only about generating new text, but reading the generations and discovering how the neural net is interpreting what you've written. I believe this particular use case hasn't been fully explored, outside of summarization algorithms.
I've heard GPT-3 could produce better text than BuzzFeed writers. You could read it without even spotting a difference.
It doesn't so much imply GPT-3 is so good (it is good, no denying that) as it implies how amazingly content-free BuzzFeed is. And mind you, there are people working on BF, writing articles and whatnot, and making pretty penny off it. Even more people read the stuff.
Maybe we the humans actually need to dilute grains of actual useful information into gobs of fluff to be able to process it well, just like vitamins have different bioavailability depending on which food they are contained within. I'm fine with this conclusion too. It's just amusing to be aware of that.
Next thing we're going to learn is that a remote-working middle manager at some large company who's rather good at his job and even has awarded annual bonuses, is discovered to have been a GPT-3 (or maybe GPT-4) simulacrum all along, with his subordinates and bosses alike being none the wiser. Which is going to say tons about the standards of management at that company.
As a non-native English speaker I would like to have a tool that points out my nonstandard usages of language. I think it could be something as simple as parsing the text and searching it against a corpus, but I have no idea how many "unique structures" a text typically contains where no one else has used those same exact words. AI could help to suggest better ways to formulate the text.
One tool I use is Grammarly. I think it’s mostly crap, but if used properly it helps you identify the more blatant mistakes. Oh, and by the way, if you need a non-native speaker ego booster, try and read native speakers’ first drafts of anything.
Yeah, that happens all the time. It always strikes me as unfair when someone comes out with an ostensibly perfect sentence and the result is "well, you didn't make any mistakes; all the words mean what you want them to mean and all the grammar is used correctly. But the sentence is still wrong; that's just not how we say that."
The correct but awkward sentences from non-natives (I also use them, being a non-native English speaker) will more easily get lost in situations with background noise, several people talking, people not paying a lot of attention, etc. because not saying things they way people expect and are used to implies extra effort to understand.
Specific examples, please? I'm curious :)
Wouldn't "piatto di brodo di pollo" translate to "a bowl of soup of chicken", as a more unnatural English sentence?
Greek is similar in that respect and I catch myself sometimes lapsing into such more micro-managed speaking, and I also noticed it in other Greeks (perhaps a few Italians also).
"Bowl of soup of chicken" is right out, but "chicken soup bowl" isn't much better. It's so anomalous that I might interpret it as meaning "a bowl for chicken soup" as opposed to "a bowl with chicken soup in it right now".
I asked my wife how to say "Happy Birthday" and she said "What you've got there is the right translation but that's not what people say".
I think that's one of the hardest things about learning another language is to not directly map words but understand what would be said in a certain context.
There is no English word for Hygge or Fika because the concept simply does not exist. You can find a nearest match but it lacks the meaning.
When you know the original intent behind a sentence (for example if you are its author), you can usually convey its meaning it in any of the languages you know well.
As your wife said, there is a way to express a friendly sentiment to somebody on their anniversary and let them know that you remember them. It's just not literally using the words "Happy Birthday".
Kind of the reverse situation than what you're describing, because I haven't learned to say those things in my native language, but for me it goes to show we learn lots of pre-baked turns of phrase that we adjust appropriately to the context when we need them.
In some countries, this is extended to even using different languages depending on the context, especially more in the Eastern countries.
on the other side, the desktop version is expensive and needs some ui work badly. they say it’s ai-based, but it feels more like a reason to drm the product than anything. still, the free version is excellent for my limited use scope, so i hope you’d have some luck with it too
the other thing to try and do is find someone that you would use as a language buddy. there’s a subreddit (i forgot the name) where people look to pair with others who are looking for/offering certain language combos, then you get on imessage/whatsapp/whatever and use each other as a language bot. right now i’m looking for someone who speaks mandarin or cantonese and needs someone who speaks english, so if that’s you op, hit me up
Actually, I find that constant repetition of the same words is absolutely necessary in technical writing. I used to do this thing where I'd try to write my research papers as if I was writing literature, trying to find new ways to say the same thing to avoid repetition, and I got some really harsh criticism as a result. I went to my university's center for academic English and they pointed out my mistake to me and suddendly it was blindingly obvious. In a technical paper you have to use the same exact words to refer to the same concept, or you will immediately confuse the reader, who will think you mean two (or more!) different things, one for each different turn of phrase. So I adjusted my writing to exploit the rythm created by the repetition of the same few words every few sentences, because of course I want to convey a precise meaning but I also want my papers to read well. And now reviewers note my papers are well written and clear; and so far nobody has complained about repetition.
my issue is that i reuse adverbs and adjectives within a few sentences of one anther and simply don’t notice it — like saying “rain fell in gentle waves,” then a sentence or two later, “the tide rolled in, soft, gentle, crashing against…” it just feels.. lazy, unimaginative to me. i read somewhere that it’s a quirk of the brain (perhaps only for some people, i don’t remember) where words in immediate memory get looped and “stuck” there. that seems to be what happens when i write, and it chaps my ass.
I'm a non-native speaker (reasonably fluent) in two non-English languages, and when I'm writing, if anything strikes me as sounding "off" then I google the phrase and see how common it is.
In my less-fluent of the two languages, I end up doing this a lot, often just to check my work. As a result I write e-mails and such very slowly, but at a fluency level exceeding my actual fluency in that language. Which seems like a good tradeoff for written communication.
It would obviously be useful to have that done for me while I type.
People used to say the same thing about Markov chains, and then the hype died down it took 20 years for another hypetrain to pull into the station. The same will happen with GPT-n.
> says very uncomfortable things about the skill of humans in recognizing genuine creative significance.
Indeed. People see pictures in clouds and omnipotent beings in the stars. Humans are uncomfortably good at finding meaning in nothings.
Even here I do wonder how well this poem would be remembered if it weren't associated with Hemingway.
So while it somewhat suggest death, it is only because of the literally context, which makes you expect something deep. If you seen that in an ad, you would not feel the same.
I'd rather think about the authors of AIs, their assumptions and their worldview.
I did like "fuzzy belly" tbf, and Kafka himself might have appreciated the purely serendipitous irony of a line about "the bugs his father used to bring him as a little boy"
Creative writing hasn't been one of the super-hyped use cases by OpenAI for the OpenAI API outside of AI Dungeon, surprisingly. For just random generation, the necessary curation can detract from the time-savings advantages. (as an aside, the API is also extremely expensive for long-form content to the point I'm not sure how the economics work for these startups even with charging monthly fees).
I'm more bullish on small bespoke models for a given use case, which is what I spend my time researching. (example: a ~1M parameter GPT-2 Magic: the Gathering card generator: https://colab.research.google.com/drive/1VOt090UzvltoBgMdUZm... )
My degree dissertation was an M:tG expert system that included a hand-crafted parser and generator for (a small subset of) ability text [1]. My Masters' thesis was a grammar induction system trained on M:tG cards [2]. And I finally managed to sneak some M:tG grammar induction in my papers as a PhD student [3].
I had a quick look at the colab of your project. If I may be critical, the examples you have are ... so-so. I've seen previous attempts to generate M:tG text with neural nets, starting with the old project in M:tG Salvation that became Roborosewater and that used LSTMs if memory serves. Results have always been a mixed bag that looks promising but always falls down holes even with hand-curation. This still seems to be the case with your project.
A couple of examples (you probably know already):
−2: Put a +1/+1 counter on each other creature you control.
+1: Each opponent chooses a creature they control. If that creature was a copy of that creature, each other player reveals their hand, chooses one of those cards, then puts it into their hand, then shuffles.
−10: Target opponent sacrifices a creature.
−9: Each opponent sacrifices a creature.
The problems I highlight above are not with grammaticality, which is certainly a big step forward with respect to the past. But many of the abilities still don't make a lot of sense, or don't make sense to be on the same card, or have weird costs etc. If you want your project to be something genuinely new, I have to say I don't think it's there yet.
My intuition is that it would take a lot more than language modelling to generate M:tG cards that make enough sense that it's more fun to generate them than create them yourself. I think it would be necessary to have background knowledge of the game, at least its rules, if not some concept of a metagame. It might also help to re-train standard NLP pipelines specifically for M:tG (e.g. some kind of ability-text specific anaphora resolution might help with "that creature" above).
Also, I note that the new online version of the game is capable of parsing cads as scripts in a programming language using a hand-crafted grammar rather than a machine-learned model [4] [5]. So it seems to me that the state-of-the-art for M:tG language modelling is still a hand-crafted grammar.
__________________
[1] https://github.com/stassa/Gleemin - unfortunately, doesn't run anymore after multiple changes to Prolog interepreters used to create and then port the project over.
[2] https://github.com/stassa/THELEMA - should work with older versions of Swi-Prolog, unfortunately not documented in the README.
[3] https://link.springer.com/article/10.1007/s10994-020-05945-w - see Section 3.3 "Experiment 3: M:tG fragment". Cringe-inducing to read just a few months after writing, sorry.
[4] https://www.reddit.com/r/magicTCG/comments/74hw1z/magic_aren...
[5] https://www.reddit.com/r/magicTCG/comments/9kxid9/mtgadisper...
As someone who did try to reproduce RoboRosewater with LSTMs years ago, I can say using GPT-2 overall has a higher signal-to-noise ratio in terms of generating interesting cards which is a win.
Are you sure? I reloaded the page once, before commenting above, but the output didn't change. I keep reloading now and the same set of cards is displayed. I reloaded with Shift-F5 to be sure ish.
>> As someone who did try to reproduce RoboRosewater with LSTMs years ago, I can say using GPT-2 overall has a higher signal-to-noise ratio in terms of generating interesting cards which is a win.
I don't know what an "interesting" card is, when we're talking about automatically generated cards. If a card looks too much like an existing card (e.g. if it has abilities that you can find on cards in a real-world set) then it's not very interesting. If it's an ability that hasn't been seen before but is only a variant of an existing ability ("Destroy target donkey") it's still not very interesting. To generate really "interesting" cards a generator must go beyond what's in the M:tG corpus, but not so far out that the abilities don't make sense anymore because that's not "interesting", just "random". The ones generated by your project are not far out enough to be what I'd call "interesting" but I think if you tried to make them more interesting they'd also become less coherent than they are currently (looking at the same few results I keep getting anyway).
(This is not very harsh criticism I hope.)
Here's some examples of curated cards generated that probably fit your description of "interesting": https://www.reddit.com/r/magicTCG/comments/n396g8/i_trained_...
On the cards you posted to reddit:
a) "Keening Vythat", "Tombsis Satyr", "Quarpling Barrier". These are typical nonsense-nonsense names (as opposed to the fantasy-nonsense names found on M:tG cards) that are typical, in my experience, of card names generated by language models trained at the character level. M:tG fantasy-nonsense names, like "Rathi", "Kapashen", "Viashino", etc, are derived from a real-world narrative created specifically to theme the cards. A backstory. Without such a backstory, "Vythat", "Tombsis", "Quarpling" are just random strings that have no reason to be used as names. This is an important limitation of this kind of card generation, especially given the very low chance that you'll see any more "Vythat" cards (and if you did, they'd likely have a completely different function). Also "Keening Vythat" sounds like a creature, rather than a Sorcery.
So, first problem: thematic consistency of names is all over the place.
b) The text on Keening Vythat is ungrammatical:
Basic lands are basic land type in addition to its other types.
d) Tombsis Satyr also has an even more interesting ability (feeds the inner Johnny). The cost and colours are right and the Angel type is just the cherry on top. Very nice. If only it wasn't called "Tombsis Satyr"... X)
e) Quarpling Barrier's ability is more of a crippling weakness than an interesting ability. A creature without Defender would not be called a "Barrier" these days, either. Pass, I fear.
f) Trade Damage is more balanced than interesting, but it's very well balanced! It would fit very nicely in a modern tempo deck. Nice also.
So that's three nice, two not so nice and a bit of a problem with themes. Still, those are not that bad at all. Unfortunately, there's just five of them. At this point it's hard to tell whether your project is able to generate better cards than other projects - and why would it, if it's using the same tech as those projects? You can fine tune and turn more knobs, but what, fundamentally, is the innovation here?
I suggest that you add the five cards above (and/or others) to the colab of your project alongside the auto-generated ones, to have examples of the "best case" and "average case" together. That'll give a boost to the appeal of your project.
Anyway, better than I originally thought but still a ways to go.
Curating only cards which are grammatically good is in itself misleading. Part of the reason I work on text generation in the first place is to illustrate its limitations.
I encourage you to run the notebook itself instead of arguing a No True Scotsman about something I never tried to assert. This notebook is not a thesis paper, just a side project about a ML model that took more time to train than to write the code for.
People who look for meaning in it will find it, but it's mostly nonsense.
It reminds me of the early face generation ... The eyes nose etc were all in the right place, but it was mimicking the appearance not the structure.
I'm more interested in a convincing paragraph than sentence.
So if AI is capable of doing some prep work, why not?
When writing fiction, my struggle is usually not in the writing itself (making sentences) but in figuring out what to write about (the content of a scene). An AI that would generate scene ideas from an input of, say, a list of characters and plot points, would be very helpful.
Outputting sentences and paragraphs, not so much (I think).
Also, the article seems to conflate Sudowrite and GPT-3 but it would be interesting to understand the difference between the two; is Sudowrite just a wrapper over the GPT-3 API that forwards requests directly? If not, what is its added value vs hitting the API directly?
We are built on GPT-3 but we have a bunch of things we do on top that optimizes Sudowrite for the narrative fiction/non-fiction use case.
For example, we have a description tool that, given a highlighted phrase, gives you options for telling phrases in 5 senses and metaphorical uses. Another tool we have analyzes a story premise and gives you possible twist and story turns, based on the genre of your story, and also gives you possible characters that could inhabit that world, complete with both external and internal character descriptions.
I agree that for writing stories, the hard part is in the forest and not the trees (although this isn't true for everyone). And systems like GPT-3 can help with that, too, simply because it's level understanding of text is approaching that of a human.
How long is the waiting list to join Sudowrite beta?
We are prioritizing folks who are working writers and have an immediate project they’re working on that could benefit from collaborating with an AI. If this sounds like you, sign up on the beta form and also send me a note about how you’ll use Sudowrite to jamesjacobyu@gmail.com
Not sure. I'm not a professional writer. I just finished my first novel (in French) and am starting a new one in English. I think the idea is great and quite original but I struggle a bit with scene ideas and general structure.
I think what I'm looking for is a sparring partner... Maybe I just need to paint a face on a volleyball...
I do way more scanning and skimming now than deep reading - authoritative source, credible, what's new, what's useful - it's all critical thinking and done as efficiently as I can make it. It feels like work. I always wonder if more structuring over the stuff that I read - entity and concept extraction, thesis extraction, separating facts from opinions, a topic-centered feed - would be better. I haven't found or built the right experience yet.
To me, narrative generating models are interesting only as a step toward a better critical reading experience, and the win there would be generating useful abstractions that help one understand faster. Arguably, writing style is one such abstraction.
Maybe there's potential in being able to generate readable narratives from things that people have a hard time describing, like datasets. Then the grail is business dashboards composed as sonnets.
Not really, no. That might be true for instruction booklet translation (they sure read like it), but no, for a decent translation, you'll need to let a human have a full go at it.
Writing is as much about filtering & tweaking GPT-3 output as sculpting is about "removing everything that doesn't look like the final sculpture", as the old adage goes.
It is useful for one purpose: To do an absolutely minimal job and to externalize some more work onto end users. It'll also be popular for spam^W SEO, and for writing filler content for content graveyards. It will not, however, be useful for writing.
I have no doubt that we'll see a lot more tooling supporting writing, and machine-assisted writing. I have equally no doubt that GPT-3 and friends aren't going to be it.
Interestingly, her job got harder when Google rolled out the new deep learning models, because instead of outputting visibly broken garbage that kind of flagged itself for checking, it now tends to produce beautifully formed sentences with subtle but often critical fuckups (wrong sense of a word, missing negation, parsing a pronoun incorrectly, etc) that can affect meaning.
That certainly isn’t the case for Chinese-English translation. It’s faster to translate from scratch than to edit machine translated text if you know what your doing and have quality standards. One of my friends did German-English translation and machine translation just isn’t for purpose there either if you have any real quality standards. You can’t skip the part where you know what’s going on.
I liked the analogy aimor made in this thread between the level readiness of autonomous driving and autonomous writing. From my experience that feels right, we're not at L5 yet - you can't just let the AI write continuously because you'll just end up with junk wrapped around some nuggets of gold.
In the future if we get to 'L5' writing, your child will have the ability to say 'OK Google, tell me a story about my adventures as a dragon slayer' and Google will respond happily with a perfect bedtime story. And the next day, you'll get a physical copy of that story in the mail. Luckily, that future isn't here yet.
The way Grim Tales was written with GPT-3 was by using some human intervention to keep the AI on track to write a cohesive narrative with a plot and dialogue. A couple of rules like removing anything that doesn't progress the narrative, and keeping the narrator in first person meant that the story was far more readable and didn't meander around like common examples.
Right now - for professional writers - AI is probably good enough as a co-pilot used to create alternate scenes at points in the narrative, or as a tool to help with writers' block.
Writing well is not easy.
I used to think I wrote well until I started reading better writers. The whole idea that we will just slap together a bunch of text using some “AI” and it will resonate with people is crazy to me. As an engineer whose dabbled with some AI and ML, and has even seen what GPT3 can do, we are as far away from being able to use AI to write a best seller as we are from landing on the Sun. Not saying it will never happen just that nothing we are building right now will be able to do that.
Good writing factors in human culture, sparks emotions in people, tells stories that resonate. I will keep going to sleep at night safely assuming that no AI will out-write humans anytime soon and you should too.
Good writing has insight. AI can’t do that yet. Most writing is filler lettuce. AI is great.
The most interesting question that it raises is whether the comment was written by GPT-3. How would we ever know for certain?
GPT-3 (and everything like it) is the perfect tool for SEO spam and filling out college papers with blather if you're confident no person is actually reading it.
It's impressive. No, really, it's an incredible simulacrum. But it's also just a slightly more practical version of the infinite monkeys with typewriters thought experiment.
Harlequin Romances, though, those little soft-core porn pamphlets for women found at supermarket checkouts, could probably be generated. There are about 4,000 of them, and they're written to a set formula - “Boy meets girl, boy loses girl on page 56, and, by page 180, the book would end with a marriage proposal.” Load up the training set and profit.
In general I don’t see utility of having AI write stuff for us to read, it’s not like we are running out of things to read. What I’m more interested in is getting an computer to memorize facts, understand the context and then subleties of a convetsarion to the extent that you could ask back questions and infer answers from that ingested knowledge we fed it. And I think we’re far from it though we’re making fast strides towards a different direction.
I never read Harlequin, but it being repetitive is not something special. Back when people read a lot of books, we did not read James Joyce and Charles Dickens exclusively. I mean, spiderman comics, ninja turtles comics, they were all the same story repeated again and again too.
We read easy for fun books.
He, ditto in Spain. In most cases the (same) authors would rehash a detective story as some sci-fi based short novel by just changing some devices and lore and call it done.
Well, in the end cyberpunk it's just futuristic noir.
An infamous crime in Chicago with mafia and detectives -> some crime in Orion with spaceships, Martian troops and so on.
Altough some of them were fairly good, like one about a time travel paradox.
We might be as far away from both. But our velocity approaching those goals could not be more different. In the last 10 years we've made tremendous progress in terms of writing. GPT-3 is leaps and bounds better than GPT-2. I'm not convinced a GPT-7 couldn't be mostly responsible for a best seller.
I like the idea of text generation as a writing aid, even a flawed network can help a stuck brain. But say writers really do hate writing, I still come back to the big question: Who's going to read it all? Cheap writing is bad enough already (think stock market news, recipe blogs, or any of my HN comments), I can't begin to fathom how much text I'll be scrolling through once GPT-7 is released. I suppose the automated writing will be consumed by automated readers then, and my involvement starts to feel kind of unnecessary.
And then there's: "While all the stars that round her burn’d, Bow’d to the ground and based their fires. To the one ever-branching cloud" When 'base' is used as a past-tense verb, we expect a prepositional phrase ... before the next period. The preposition is never 'to'. Doesn't matter; the period stopped the English part.
"That blew and drifted—blow and drift;" What?
"If you told me that Coleridge wrote it, I would believe you."
NExt.
Due to an obscure low-level technical decision (encoding text into BPEs), phonetics are erased from the GPT training data, and so therefore neither GPT-2 nor GPT-3 can learn to rhyme or make puns (they can only memorize relatively common pairs by bruteforce): https://www.gwern.net/GPT-3#bpes
I put a good deal of effort into testing this, including "Kubla Khan": https://www.gwern.net/GPT-3#samuel-taylor-coleridge Just doesn't work, and it never will until OA trains a model using some sort of better encoding which preserves phonetic information. (Character-based, unigram LM, or BPE dropout might all work, we don't know yet.)
This kind of help with the mechanics of composition and usage, however, is a far cry from the critical thinking needed to be an effective advocate, expositor, or entertainer. There’s more to being a writer than making plausible sentences and paragraphs.
Imagine. You could feed it all of Tolkien's writings and tell it to give you an epic saga of novels set in the First Age of Middle Earth. You could tweak a parameter and instantly get a version of your favorite novel that proceeds with a different main character or a different choice and have it be indistinguishable from a version written by the original author and one hundred percent consistent with events and previous plots.
I'm not too ashamed to say that if memory-erasure technology existed I'd be sorely tempted to experience my favorite media for the first time over and other. Creative AI is the looming spectre of that danger, as it becomes more and more skilled at giving us the best, we will grow intolerant of anything less.
Arguably, you are describing about 80% of the Fantasy genre, only it's all written by humans.
Sorry if that sounded like a "shallow dismissal" as per the HN guidelines. I used to love F/SF and then I found I couldn't read it anymore because it all felt like the same few story elements were reused over and Over and OVER again. I get the same feeling of dread boredom when I browse the Fantasy isle at bookshops today. <evil power> is rising in the <cardinal direction>. <Hero> must <undertake heroic quest> to defeat <evil power>. It all got so formulaic you could write a Cluedo variant based on it: The Orc Captain with the Great Axe in the Dark Tower.
This, btw, was all before I (very belatedly, because I'm not a native English speark and so I read only what was translated to Greek) discovered Terry Pratchett, Jeff Noon and Iain M. Banks, who were a breath of fresh air (unfortunately rarely refreshed since then) and yeah, as a kid, I read a lot of F/SF. Explains my school grades, hah.
Do you think that such tools will have the eyes of the regulators on top of them due to copyright issues, since the data used could have a lot of copyrighted material?
I haven't yet made my mind about this: is the output original content, or just shuffled proprietary content?
1) The author inadvertently uses output from GPT-3 that happens to be close enough to copyrighted material to be a problem.
This is why we recommend to authors do two things: a) enter as much of their own original work to prompt Sudowrite and b) edit and incorporate the AI output organically into their work. The chance that GPT-3 outputs verbatim copyright work in that case is small. As we grow, we also plan to incorporate plagiarism checks in the background.
2) Someone could challenge the author’s copyright on a piece of work that was primarily written by GPT-3.
This has not been tested in the courts, and I suspect it will come to head in the next few years. For me, the question of who owns copyright is dependent on how much work the author has done to the AI output (there are some parallels to the Monkey Selfie case [2]). For example, Stephen Marche (the author of the new yorker post from OP) also wrote a short story [1] using Sudowrite, in which he claims that 17.1% of the final words were from the AI. But of course, he has done non-trivial amounts of work to edit and incorporate those words and made it his own. So in this case, he def deserves ownership. All of our Sudowrite authors use the AI similarly in their process.
However, what if an author uses vanilla GPT-3 with no humans int the loop to generate hundreds of biographies of famous people based on wikipedia entries? I don’t have a good answer to this. It shouldmay come down to how much the author has done to guide GPT-3 and/or pre-post-process it.
Here’s further reading:
http://rutgerslawreview.com/wp-content/uploads/2017/07/Rober...
https://core.ac.uk/download/pdf/77599763.pdf
[1] https://lareviewofbooks.org/article/the-thing-on-the-phone/
[2] https://www.npr.org/sections/thetwo-way/2017/09/12/550417823...
It has no clue what it's talking about, of course. It's terrible at fact-based material, because what comes out looks plausible, but contains made-up facts. Some facts will be correct, because they were mentioned a few times in the training database. Others are just plausible words strung together.
What it really demonstrates is that much of literature has less behind it than was previously supposed.
It depends on if you think writing ability is more like the top graph, or the bottom graph. If it's like the top graph, humans have plenty of time to adopt AI tools, and incorporate them into the writing process.
If it's like the bottom graph, then GPT-4 comes out in 12 months and is human equivalent, and GPT-5 comes out a year after that and is a better writer than any human could hope to be.
A page is good enough for the clickbait industry.
It reads as good or better than a human would do. Maybe not better than the BEST human marketer would do, but far better than average, and instant + free(ish).
I'm uncomfortable with this statement because GPT-3 is in a sense copying or sourcing a massive corpus of human literature to use as the paint on its brush. If it can make a sentence that sounds Shakespearean after performing a statistical analysis of all sentences ever written by Shakespeare and using fancy averages and weightings of those sentences, how does that diminish the value of Shakespeare? In a wider sense, isn't that what's happening, but instead of using every sentence written by Shakespeare the goal is to use every sentence ever written?
That said, how GPT-3 and its descendants affect the market value of "literature" is... sorry I can't resist... a story yet to be written.
> Gupta imagines the product turning into a resource that writers will pay fifteen to twenty dollars per month to use.
That seems like a great deal to sublet some portion of all those resources.
For homework, you can calculate teraflops per second per second, which would measure the acceleration of teraflops.
Seems contradictory to everything I've heard about the iteration loop between practice, play, and discovery that characterizes mastery in any craft, be it programming, writing, music, sports, and so on.
It doesn't.
Is it still GPT-2 ?