Hacker News Reader

Comment by zerof1l | Hacker News Reader

One big issue with all those NN-generated texts, be it GPT3 or this, is that it's just a collection of seemingly random words.

I think that the issue lies in that we approach content generation in the wrong way: a sequence that goes from start to end in one go. The idea is at the start and by the end of the sequence, the model had moved far away from the original idea. We humans, start with one or a couple of ideas and we iterate over them many times and then expand them in all directions. If only a model could do that.

View on HN

I think the real problem with GPT3 etc is it's not really original.

It's a corpus of text so large and varied, that simple requests can bring up a few good examples, which can be mashed together to seem new. It's essentially a smart search & plagiarism process. I suspect if you could view the main contributing sources it would be less impressive.

for example, I was really impressed that GPT3 could write code - until I realised that it contained most of stack-overflow (including code examples) - no actual inference was happening, it was just search.

On the one hand, there always seemed something kind of flawed about the work OpenAI is doing versus DeepMind.

On the other hand, people talking about OpenAI all the time helps it thrive - all publicity is good publicity. So the best thing you can do personally is, if you don't like what they do, just don't talk about it anymore.

This is a valid literary technique, if you're not aware.

https://en.wikipedia.org/wiki/Cut-up_technique

It's a know technique, I wouldn't say that makes it "valid". It also seems to be a inspirational technique i.e. adding an artificial limit to spur creativity.

The text is ultimately managed by a real human being otherwise capable of writing original text. That human being hasn't been exposed to English entirely through the corpus of test given, as would be the case of GPT, which has no previous knowledge of language.

TLDR; the technique gives the text to an already create human being, rather than expecting creativity to arise from the technique itself.

I've had to explain this to non-techy family. GPT3 is still a monkey at a keyboard. Only, the keyboard is huge, and each button is a whole phrase. Maybe the keyboard reconfigures itself after each keypress, to make the phrases scan a bit better. But it's still a monkey at a keyboard.

That's not true, or is at least making GPT3 out to sound more trivial than it is. I've spent hours "interviewing" people about dragon sightings in Washington D.C., or Elon Musk about a fictional startup, or similar. I've had it generate startup ideas that are definitely original and elaborate on them, including company names that are concatenations of words that have never been used as a proper noun. I had it generate an interview with the founder of a fictional 1980s video game company, and it went into detail about the series of fictional games. It was all plausible, but the names were unique strings that I could not find on Google.

I'm not impressed by fictional nouns, be they people names or fictional corporations. GPTs special sauce was supposed to be text by, at the least, the paragraph, so it's the "generate/elaborate on startup ideas" and "went into detail about the series of fictional games" I'd be interested in.

Without seeing the source material, how original do you know these generations to be?

> I think the real problem with GPT3 etc is it's not really original.

Everything's a remix.

Reply on news.ycombinator.com