Comment by QasimK | Hacker News Reader

Comment by QasimK | Hacker News Reader

I’m continually amazed - flabbergasted - by GPT-3. I’ve read stories, articles and HTML written by it and each time I am shocked at how good the output is. This essay made me laugh!

It’s practically indistinguishable from a human. Not a creative, insightful and unique human. But an average human? Yes, I cannot tell the difference.

I must repeat that - I cannot tell the difference!

This can probably completely replace or supplement most online content that I see including news, certainly on the vacuous side of things of which I think there is a lot of content.

Those online recipes with irrelevant life stories before them? Replaced. Those opinion pieces in news? Replaced. Basic guides to tasks? Probably replaceable.

I know I probably only see the best output, and it would be nice if I had more context, but the peak performance is amazing

The twitter video showing GPT-3 generate HTML based on your request? I think there’s a lot of potential. I don’t knew whether it can, in general, live up to these specific examples though.

If you pay just a little attention, you absolutely can: GPT-3 is not saying anything. Even the 'lowest' humans are usually trying to communicate something when they are telling a story, or teaching you how to do a basic skill, or giving you directions.

GPT-3 can't do any of that. It can pick up clues from the text to produce incredibly realistic sentences that are related to the broad topics of some text, but it's all smoke and mirrors in the end - there is no model of the world getting expressed in communication, it is just mindless aping of similar speech.

And yes, this basic skill that GPT-3 has is enough to replace some human tasks, like inventing plausible sounding stories for a recipe or perhaps even taking news from one site and writing them on another with slight alterations. Perhaps it will even be able to take some facts and weave them into a speech about that topic.

But it is not even close to doing something like real journalism, even at the level of a car mechanic telling you what happened down at the mall.

GPT's text as quoted in the article has a clear thesis stated upfront, later expands on it with examples, summarizes its arguments at the end, and does all this with gusto.

You cap your comment by a non-sequitur that GPT is not going to replace journalism.

IMO the piece of text generated by GPT offers more insight and is wittier than yours.

My comment was not an essay, it was a response to someone else's comment. The GP was explicitly saying that GPT-3 could probably produce many of the news content they read, and I was replying to that.

Please also note that it's not very clear to what extent the text of the article is edited - the non-bold text is written by GPT-3, but I don't think it was produced as a single block of text. Instead different parts (sentences? Paragraphs?) were produced individually, selected by a human from many other responses, and assembled together in the shape we are shown. The train of thought among the paragraphs is most likely entirely human work, not AI work, and only the best sounding paragraphs out of a lot of gibberish were likely selected.

It would also be interesting to see how close those paragraphs are to something in GOT-3's training corpus, in terms of structure if not explicit language.

> And yes, this basic skill that GPT-3 has is enough to replace some human tasks, like inventing plausible sounding stories for a recipe or perhaps even taking news from one site and writing them on another with slight alterations. Perhaps it will even be able to take some facts and weave them into a speech about that topic.

> But it is not even close to doing something like real journalism, even at the level of a car mechanic telling you what happened down at the mall.

You're just talking about layers of abstraction. GPT-3 works with chunks of 3 letters. It can now.

Combine chunks to form valid words. Combine words to form syntactic sentences. Combine words to form semantic sentences. Combine sentences to form consistent paragraphs. Combine paragraphs to form trains of thought.

It can't combine trains of thought to produce a consistent point.

But it's already working successfully at 5 or 6 levels of abstraction up. I don't think it's that much harder to get one to two levels higher in abstraction. It doesn't know level 7 is any different from level 6. It just needs the data and compute to start modeling at that level.

In the past you could've convinced me a different architecture is needed to accomplish that. But I also wouldve doubted it could get this far - why would something stop it now?

It just needs to study more long form work and how to reach a conclusion in an essay starting from paragraph 1.

Define "semantic" and "trains of thought."

Personally I'd label this human satire, not raw GPT-3 output.

I'll change my mind if I see evidence that proves that's incorrect.

I feel like I can understand it (GPT-3 generated text) better than meme-oriented Reddit callback threads. Which may have more to do with burying meaning more than explicating it.

> If you pay just a little attention, you absolutely can: GPT-3 is not saying anything.

Check out Figure 3.13 in the GPT-3 paper, https://arxiv.org/pdf/2005.14165.pdf.

The authors experimented with 200-word news articles, to see whether 80 human judges could tell the difference between human-generated and GPT-3-generated ones.

It turns out they could not: the human judges correctly identified GPT-3-generated content only 52% of the time, essentially as good as random guessing. (And no, the machine-generated articles were not cherry-picked for the experiment.)

This is a good counterpoint.

I do wonder though how close the articles that GPT-3 produced were to the articles it had been trained on. For example, the Methodist Church Split article that it produced has a lot of very specific facts about the Methodist Church and about the split, which shows that it had texts about that spefic event in the training set.

It also has a sentence which contains a pretty obvious non-sequitur, but it's easy to miss it or assume that it's a mistake that a human made.

So overall, I'm guessing GPT-3 may actually be pretty decent at re-telling a story with different words, which sometimes is very hard hard to distinguish from a human doing the same thing.

They also don't describe the way they programmatically selected the output, though I am willing to believe that they more or less randomly sampled the output from each model.

>> The authors experimented with 200-word news articles, to see whether 80 human judges could tell the difference between human-generated and GPT-3-generated ones.

I disagree with tsimionescu that this is a good counterpoint. The comment you reply to says that "GPT-3 is not saying anything". The figure you refer to shows that human judges could not tell the difference between human-generated and GPT-3 generated text. That's apples and oranges. That some humans weren't able to detect autogenerated text doesn't say anything about whether the autogenerated text said anything.

However, the comment also said this is a way to tell GPT-3 text from human text.

> If you pay just a little attention, you absolutely can: GPT-3 is not saying anything. Even the 'lowest' humans are usually trying to communicate something when they are telling a story, or teaching you how to do a basic skill, or giving you directions.

you're right; this AI is no substitute for a journalist. however, I do think the "essay" compares favorably with some papers I peer-reviewed for my college writing seminar. sometimes humans really aren't trying to communicate anything; they are just trying to hit the minimum word count and get a passing grade.

Sure, as I said, there are human tasks that may appear creative but aren't.

Also, please note that the 'essay' is assembled by human selection from GPT-3 produced output (whole paragraphs?).

> GPT-3 can't do any of that. It can pick up clues from the text to produce incredibly realistic sentences that are related to the broad topics of some text, but it's all smoke and mirrors in the end - there is no model of the world getting expressed in communication, it is just mindless aping of similar speech.

You know this inevitably tempts the cynical question of how different what you describe is from (to put it optimistically) clickbait generators and (to put it still more cynically) much of the content generated today ….

It may or may not be smoke and mirrors, but there's a clear structure to the essay, as well as a logical structure to the arguments in the essay. For example, "humans think they're intelligent", "humans are wrong about everything", and therefore "intelligence isn't about being right".

It even concludes with reasonably good advice: to pass a Turing test, AIs should say things that are true, and tap into human emotions. To me, this disproves the idea that it's "not saying anything". It's definitely saying something, and that something is both true and not commonly understood by the general public. It is therefore capable of "teaching a basic skill".

I find that very impressive, and I'm surprised that so many others here don't.

As I stated elsewhere, the essay is almost certainly constructed by the human writing the article, out of cherry-picked output by GPT-3 stitched together (I'm guessing at the paragraph level).

Also, again almost certainly, the information about what it takes to pass the Turing test is taken from some text in the corpus it was trained on. What GPT-3 did do is recognize that that is relevant to the topic of AI vs human intelligence, but not much more.

> Those online recipes with irrelevant life stories before them? Replaced.

Oh man. Machines replacing humans writing stories that humans were only writing to satisfy machines in the first place.

I think the hurdle is going to be getting it to write accurately. For example, if you want it to write a news article, ideally you want to give it some facts (this event happened, x people died) and it pumps out some accurate prose about it. I'm not sure you can trust GPT-3 to do that. It could produce something that sounds good, but there is not really a guarantee it is not spewing falsities that came from it's training data.

I don’t trust most journalists today to pump out accurate prose around the facts.

Some GPT-4-like system may be a better writer than almost every journalist but have many more blatant errors or incoherent deductions.

Edit: Or perhaps it will actually develop a rudimentary factual understanding of the world...

I can't tell these examples from human either. I half suspect that Arram Sabeti is running a genuine Turing Test and that many of these outputs were actually written by humans.

I haven't done a survey of GPT-3 output from other sources, though, so I don't know how typical these outputs are. To those who have: what is your opinion? Do you think they're real (in the sense of being written by GPT-3) or fake (in the sense of being written by a human)?

This one seems a bit to good to me to be GPT-3, but it does have some of the general GPT-3 signs.

The tone, the use of voice, the straying abstract line of thinking. At the same time these flaws could all be faked.

Overall I give the author the benefit of the doubt that they are legit. But if I had to pick an essay that's fake that I've read (knowing one had to be fake), it'd be this one.

But safest best is that it's real, and we're just amazed by how good gpt-3 is.

Do you really feel this was written by a human?

If I read this without the context, I would only assume it was written by a human given that is the implication when I read something like this, but I would immediately think it's idiotic and move on.

There is simply nothing being said here. It is just like a desperate student going online to copy sentences, phrases, and words from articles and paste them into an essay, which at its core, is basically what this "AI" is doing.

It reads like a witty, sarcastic take on the attempts to determine what would make a computer intelligent by turning it on its head.

Frankly, with a little bit of framing, it'd make a great sci-fi short story.

Yeah I thought it was a rhetorically framed as GPT-3 but really written by a human writing in character as an offended strong AI calling humanity on their bullshit and tautologies. They emphaisze successes but failures dominate and the only consistency is chauvanism in only defining in likeness to success instead of anything relevant like being correct. The simplicity and weakness of the failures listing gave more of a "the AI is outright trolling humans by using patronizingly sophmoric examples" akin to a peeved climatologist starting off with the basics of the water cycle when explaining how global warming can lead to both more flooding and droughts.

GPT-3 has finally reached the level of idiotic human! That's actually quite a large step forward from any previous language model.

Reply on news.ycombinator.com