Comment by throwawaygh | Hacker News Reader

Comment by throwawaygh | Hacker News Reader

Some of the criticism in this comment section is completely fair — the authors are providing exactly the type of prompts that GPT-3 breaks down on and some of these examples might be cherry-picked continuations. And the authors do have personal interests at stake. (NB, the exact same criticism is true about a lot of articles lauding GPT-3, which is why public discussion of GPT-3 in general is such a dumpster fire.)

So, other than “GPT-3 isn’t an AGI” [1], I’m not sure what to take away from this article other than the actual substantive criticism is at the beginning of the article:

“[We have previously criticized GPT-2.] Before proceeding, it’s also worth noting that OpenAI has thus far not allowed us research access to GPT-3, despite both the company’s name and the nonprofit status of its oversight organization. Instead, OpenAI put us off indefinitely despite repeated requests—even as it made access widely available to the media... OpenAI’s striking lack of openness seems to us to be a serious breach of scientific ethics, and a distortion of the goals of the associated nonprofit. Its decision forced us to limit our testing to a comparatively small number of examples, giving us less time to investigate than we would have liked, which means there may be more serious problems that we didn’t have a chance to discern.”

Several other researchers I know — very good researchers who happen to have been publicly critical of GPT-2 — have not been given access.

This isn’t how science is done (access for reproducibility and probing, but selectively and excluding prominent critics). If any other company behaved like this no one would take them seriously. Or would at least temper every “wow this is amazing” comment with “but the community can’t really evaluate properly, so who the hell really knows”.

--

[1] given misunderstandings down-thread, and to be clear, this is a tounge-in-cheek sentence fragment meant to emphasize that "the article doesn't tell us anything else we didn't already know". Obviously, neither Open AI nor Marcus claim that GPT-3 is an AGI.

For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. This could realistically only occur if they cost-effectively fine-tune away the more egregious problems in beta - or convince enough investors that their next model with a 100 million dollar price tag will be able to handle something approximating AGI for realistic applications.

This is the same game plan that Self-driving car companies have been playing. The product is only an investment round away, if we just happen to spend more money on bigger models using more data. This will either end with a price tag in the billions that investors are unwilling to pay, or successful monopolies. Allowing additional researchers to perform extensive analysis of the technique is likely to just reveal systematic flaws which increase the risk that the next round of research will produce a successful product, or limit the companies ability to create a monopoly following success.

This isn't necessarily a bad thing for advancing the state of the art, but it does introduce a whole lot of BS into the current state of research.

~50-300 million sounds like an over-estimate, and the investment in GPT-3 is miniscule compared to SDC.

Plus, GPT-3 generates value even without direct product impact. I'm sure MSFT sales reps are already folding tons of nonsense about gpt-3 into their Azure pitches. Industry labs like OpenAI and DeepMind replace/augment the "Research & Development" model with a "Research & Marketing" model.

Seems spot-on. One trick to estimate a startup's burn rate is to multiply their number of employees by $200k. It's not too accurate, but it's within the ballpark.

So how many employees does OpenAI have? Supposing they have 500, that's a burn rate of $100M/yr.

250 employees, $50M/yr.

100 employees, $20M/yr.

Gpt-3 is just one of openai's many projects. Theres ~20 authors on the paper, and they all almost certainly did not spend a full year on this project. So 300m is completely wrong.

I believe significant share of GPT-3 cost is machine-hours that were spent training this model - months of hundreds top-tier NVidia machines.

Edit: estimates range from $2-$5MM to $15MM

https://www.reddit.com/r/MachineLearning/comments/hwfjej/d_t...

I thought MS was giving them Azure GPU instances for free?

“Free”, which is basically Microsoft, as an investor, has paid those $4-$15MM minus ~30% of margin they make over pure operational costs.

Electricity was burned and NVidia GPUs that could have been doing paid job were doing GPT-3 training instead.

$15mm is a rounding error for MSFT

> For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model.

On top of that, does anyone have an idea for what practical applications the model could be used? So far I've only seen the model being used to confuse people; how would one turn that into an ethical business? It seems to me that the "BS route" is indeed the logical course.

Based on the fact GPT-3 seems capable of producing flowery language that is confusing and ultimately nonsensical, it seems to me that GPT-3 has a future writing speeches for politicians ;)

I think this is why GPT-3 scares VC Twitter so much - nobody can tell the difference.

Entertainment. e.g. $3 per month for My Virtual Friend.

Before you scoff, consider that Pet Rocks were once a (profitable) thing.

Eliza, despite its limitations, sparked considerable engagement with those who were willing to chat with it at length without derailing it.

https://qz.com/1439200/loneliness-costs-the-us-almost-7-bill...

Eliza comes up a lot in these discussions. I’d like to know if would have the same results today, now that chatbots, video games, phone robots, etc. have given millions of people the mental model of getting responses from a box with no thoughts. I expect it would very different from running the experiment at a time when the only things that could talk back when talked to were other people.

Eliza (seemingly) implements a rudimentary conversational model of psychotherapy that facilitates transference.

https://en.wikipedia.org/wiki/Conversational_model

This model differs from goal orientated transactional conversational models e.g. phone robots in so far as the objective is not to direct behaviour, but rather to facilitate self-expression, a task which can be performed with sparse requirements for real-world logical semantic modelling.

Eliza could always keep within the bounds of transference by deflecting back to interlocutor e.g.

You said earlier that ____________

Are you worried about __________

Tell me why not?

Do you like talking about __________

Anything specific?

Let's talk about something more interesting.

- - -

Well, not always. Eliza was easily derailed into revealing its logical semantic paucity, thus breaking the spell of transference.

GPT-3 has reached a significantly higher plateau of semantic pseudo-competency, but what if it were applied simply to the application of transference conversation? So long as it allows lonely people to chat away without breaking the spell, that might be enough for GPT-3 to earn $3 a month.

Could there be a market for friendertainment?

https://phys.org/news/2018-12-americans-lonely.html

> Could there be a market for friendertainment?

Aren't there tons of English-speaking people in third-world countries capable of being worthy pen-pals and filling this market gap?

GPT needs to be capable of at least the following in order to be a viable virtual companion: Memory, Reasoning, Metanarrative, Emotion, Empathy, Intent and Personality. GPT is not, and likely will never be a good conversational agent. No purely neural network based approach will, conversation is not a field you can fit a model to and hope to get something that works.

I actually just sent out an API request for a particular application that I believe GPT-3 should be capable.

I'm a university mathematics instructor. I find that a lot of students struggle with doing proofs and I think GPT-3 can help with that. A proof is essentially a sequence of logical inferences. I believe that, when given a proof written in natural language GPT-3 should be capable of detecting the logical implications made within. From there, it would be possible to correct student's mistakes or help them improve their answer.

Really? This seems like the exact sort of thing GPT-3 would be very bad at. The statements aren't the product of an internally consistent logical model, they are statistically plausible word sequences learned in a really clever way from a huge corpus of text.

That said, before dismissing it sight unseen there are effective and quick ways to test its powers with mathematics. You could prime an instance with the axioms defining a group, or a topology, and ask it to produce proofs of varying difficulty from intro abstract algebra or topology textbooks. Whatever the answer it would make an interesting blog post. What happens when you ask it to generate repeated outputs for the same prompt? What happens if you ask it to prove an untrue proposition?

So perhaps it could produce convincing looking proofs that are ultimately nonsensical.

It could generate math problems and have students try them out. Might be a bit of fun.

That's probably precisely an area you would NOT want to work with GPT-3. As Julian Togelius put it:

> GPT-3 often performs like a clever student who hasn't done their reading trying to bullshit their way through an exam. Some well-known facts, some half-truths, and some straight lies, strung together in what first looks like a smooth narrative.

https://twitter.com/togelius/status/1284131360857358337?s=20

Maybe in games where you want more convincing NPCs, something like Skyrim, where the main characters are very well branched, but it’s impossible to give every one depth.

This is one application of GPT that no one would mind ethically. Have a person oversee the logical/contextual structure.

I have a few ideas for disability aid applications.

I wonder if you can get GPT-3 bots to spam Reddit, Twitter, and Facebook into oblivion. I also wonder what percentage of users would notice.

Give them a political bent - that's probably what the state actors are trying to productionize right now. Target posts with a sentiment that disagrees with yours, then make the bots follow those users and inundate them with replies wherever they go. Hell, even brands might step in and start doing it.

What's the value of a social network when 25% or more of the comments are from GPT-3 bots?

GPT-3 doesn't need to know anything at all for it to have a very noticeable impact on the web and social media.

I've been wondering about the exact same thing. Basically, a sufficiently calibrated and targeted GPT3 bot swarm could be employed to render at least some parts of these communities into useless echo chambers.

I think that while GPT3 posts are usually identifiable after reading some sentences, I often also find that it's harder to realize when consuming precisely this kind of social media. I often just consume it absent mindedly as opposed to reading a real article.

With most comments being a paragraph or less, I think social media is done for unless it finds a way to put up a wall.

One thought is the subscription model of older social media sites, such as Something Awful. Pay $5 to get an account. Break the rules, get banned, pay another $5.

That's a lot of friction for your most valuable consumers, though. Marketers want users that are less discriminating.

Another is a shift to video and multimedia-based social networking. TikTok, Instagram, Twitch. It doesn't save the comments, though. I'm not sure how long we've got before the videos themselves can be generated with a high degree of novelty.

AI is going to change the game significantly. We're perfectly timed for a technological change of winds that enables new upstarts to challenge the incumbents. I'm kind of excited.

I don't now about that. I believe if you had it calibrated correctly no one could tell the difference between a single GPT-3 comment and a human comment.

Maybe after a bit of dialogue you'd have a higher chance, but even then I suspect (from playing around in AI Dungeon) that GPT-3 could do very well.

The GPT-3 paper goes into this. They consulted experts:

The assessment was that language models may not be worth investing significant resources in [by state actors] because there has been no convincing demonstration that current language models are significantly better than current methods for generating text, and because methods for “targeting” or “controlling” the content of language models are still at a very early stage.

The paper has more details.

Very true. An example I just saw (and reported) of something that's about to get much worse:

https://rennlist.com/forums/987-981-forum/1177333-best-oil-f...

("I have Tucson 2.0 and I can say that the mechanisms are as similar as possible and it was also very difficult to pick up oil, because it is also old. That is, hard oil will not fit there, maybe even spoil the engine and transmission. Therefore, it is better not to take risks and take exactly those oils that are best suited to such types of cars and engines. You can also read about the best oil for tucson 2.0 (spam link redacted), there are different variants of such machine oil described in details. I personally took Mobil 1 Extended Performance 5W-20 Full Synthetic Motor Oil and I do not regret, because of its full synthetics it is quite loyal to the engine. Therefore, it is ideal if you have an old car, but in any case, study the top and then already select from it.")

This is pre-GPT quality material, and it's already good enough to hang around unreported for days in some cases. A lot will depend on how much effort forums are willing to put into vetting new users.

I wrote about that scenario last month: Commentdämmerung (twilight of the comments)

https://simonsarris.com/commentdammerung

> Several other researchers I know — very good researchers who happen to have been publicly critical of GPT-2 — have not been given access.

Wow, this is an incredible nasty move. This is allow telling about the confidence they have in their model.

OpenAI never claims GPT-3 is an AGI, do they? If they don't, why should they pander to people who criticize it for not being one? Are there other claims they make, that have been/could/should be refuted? It's simply the most advanced text generator by far. Nothing more, nothing less.

> pander

The goal should be pushing science forward, not maximizing brand value. Enabling top scientists in the field to reproduce and probe your work is not "pandering". It's participating in the scientific process.

It’s easy to sit here and say what their goals “should” be, but it doesn’t change the fact that they never claimed it’s an AGI.

Gary Marcus doesn't claim that they claim that it's an AGI, so I really have no god damn clue what y'all are going on about here.

Clearly, the point I'm making in the top comment is the following: OpenAI apparently restricts access to people -- even very highly respective scientists -- who happen to be critical of their previous work. It's impossible to prove intent, but limiting access for people like Gary Marcus is while giving access to tech journalists likely to write breathless hype pieces is odd behavior that deserves to be criticized.

From a quick search on Gary, he's, in essence, spreading the word that GPT-3 isn't an AGI. Maybe this is useful to laymen to quench fears that it is one, but how should this garner a relationship with OpenAI? There are still people who have access who admit it's not, including OpenAI themselves already. I do wish the model was truly published openly, but despite it not being an AGI, it's already powerful enough to be potentially used for nefarious purposes. I do think there's an argument to be made that we should try to keep it in a good light to further interest in the field. How many more articles do we need like the one linked by OP?

> I do think there's an argument to be made that we should try to keep it in a good light to further interest in the field. How many more articles do we need like the one linked by OP?

This is how winter happens and is a decidedly anti-scientific mindset.

Do you consult with non-technical/low-technical execs on AI investments? If you did, the answer to your question would be obvious. For every dollar Microsoft and VC firms can spend on "we're the best technical minds in the world" Research&Marketing departments, there are thousands dollars in the rest of the economy that can only be captured by ensuring execs make prudent AI investments. Step zero is always deconstructing the hype around GPT/Alpha*/etc.

Is a winter not a widespread lack of interest in the field? I would stay step 0 is getting people interested by showing what's possible, and step 1 is showing what's not currently possible but should be.

A widespread lack of interest caused by a failure of a previous technology in the field to live up to it's hyped claims.

> Gary Marcus doesn't claim that they claim that it's an AGI, so I really have no god damn clue what y'all are going on about here.

Well, who is Gary Marcus anyway? I looked up his publications in the last 2 years on Google Scholar and I see no hard contributions, all he has are some publications about futurology and AI critique. His wiki page says he's a cognitive scientist who once sold an ML company to Uber, but not an AI expert.

Why didn't Gary invent a better language model to show us how it's done? If he knows better than the guys at OpenAI, let him show the path ahead. When someone has superior results it's not constructive to throw bullshit at their accomplishments.

Yes, it's easy for anyone with a modicum of knowledge about scientific integrity. Wonder why OpenAI doesn't seem to get it.

Look, they're probably getting millions of requests. At least some of those people who are respected scientists in NLP/AI but cannot get access were almost certainly just over-looked. It was really seeing that even Gary Marcus and Ernest Davis cannot get access that pushed me over the line on this one from "be gracious, moderating access to a finite resource is hard" to "wow this is sketchy".

> Look, they're probably getting millions of requests. At least some of those people who are respected scientists in NLP/AI but cannot get access were almost certainly just over-looked. It was really seeing that even Gary Marcus and Ernest Davis cannot get access that pushed me over the line on this one from "be gracious, moderating access to a finite resource is hard" to "wow this is sketchy".

You seem to have changed your stance 180° in the middle of your comment.

Maybe the person you were replying to is actually a GPT-3 comment bot and we're already screwed.

He didn't, he is saying that the previous comment anecdotal evidence could be easily explained by the amount of interest into the project by many people, which can't all get access to it, but then show how this same explanation shouldn't apply to Gary Marcus and Ernest Davis.

I had a belief (assume best intentions unless there's evidence otherwise). And then, later, I modified that belief given some new compelling evidence.

Are you saying that you modified your belief while writing your comment, or that you rewrote your posted comment after changing your mind?

Yeah, me too. I was really starting to be convinced of GPT-3's potential but this is a major red flag for me. Are they researchers or businessmen?

This is the first time I've heard this. This kind of thing definitely needs more news coverage.

GPT-3 is what it is no matter who got invited to the beta. I have personally tried it and it clearly is above any language model up to date, and made me seriously consider prompt programming as an emerging field.

What I would like to see next is: to extend the corpus with task specific data where available, making it more performant on these specific tasks; to have a searchable memory module (ability to retrieve text from its corpus or new text additions); to enlarge its context window; to enlarge the corpus, including more non-english content; to be trained on more than just text: images, audio and video, for grounding. It also needs a leash - a second model to stop it from generating profanity or insulting things and better ability to control the expected size and format of the generated text: dialogue, article, novel, etc.

>> And the authors do have personal interests at stake.

Can you say what those are?

Reply on news.ycombinator.com