Comment by lumost | Hacker News Reader

Comment by lumost | Hacker News Reader

Parent Full thread

For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. This could realistically only occur if they cost-effectively fine-tune away the more egregious problems in beta - or convince enough investors that their next model with a 100 million dollar price tag will be able to handle something approximating AGI for realistic applications.

This is the same game plan that Self-driving car companies have been playing. The product is only an investment round away, if we just happen to spend more money on bigger models using more data. This will either end with a price tag in the billions that investors are unwilling to pay, or successful monopolies. Allowing additional researchers to perform extensive analysis of the technique is likely to just reveal systematic flaws which increase the risk that the next round of research will produce a successful product, or limit the companies ability to create a monopoly following success.

This isn't necessarily a bad thing for advancing the state of the art, but it does introduce a whole lot of BS into the current state of research.

> For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model.

On top of that, does anyone have an idea for what practical applications the model could be used? So far I've only seen the model being used to confuse people; how would one turn that into an ethical business? It seems to me that the "BS route" is indeed the logical course.

Based on the fact GPT-3 seems capable of producing flowery language that is confusing and ultimately nonsensical, it seems to me that GPT-3 has a future writing speeches for politicians ;)

I think this is why GPT-3 scares VC Twitter so much - nobody can tell the difference.

Entertainment. e.g. $3 per month for My Virtual Friend.

Before you scoff, consider that Pet Rocks were once a (profitable) thing.

Eliza, despite its limitations, sparked considerable engagement with those who were willing to chat with it at length without derailing it.

https://qz.com/1439200/loneliness-costs-the-us-almost-7-bill...

GPT needs to be capable of at least the following in order to be a viable virtual companion: Memory, Reasoning, Metanarrative, Emotion, Empathy, Intent and Personality. GPT is not, and likely will never be a good conversational agent. No purely neural network based approach will, conversation is not a field you can fit a model to and hope to get something that works.

Eliza comes up a lot in these discussions. I’d like to know if would have the same results today, now that chatbots, video games, phone robots, etc. have given millions of people the mental model of getting responses from a box with no thoughts. I expect it would very different from running the experiment at a time when the only things that could talk back when talked to were other people.

Eliza (seemingly) implements a rudimentary conversational model of psychotherapy that facilitates transference.

https://en.wikipedia.org/wiki/Conversational_model

This model differs from goal orientated transactional conversational models e.g. phone robots in so far as the objective is not to direct behaviour, but rather to facilitate self-expression, a task which can be performed with sparse requirements for real-world logical semantic modelling.

Eliza could always keep within the bounds of transference by deflecting back to interlocutor e.g.

You said earlier that ____________

Are you worried about __________

Tell me why not?

Do you like talking about __________

Anything specific?

Let's talk about something more interesting.

- - -

Well, not always. Eliza was easily derailed into revealing its logical semantic paucity, thus breaking the spell of transference.

GPT-3 has reached a significantly higher plateau of semantic pseudo-competency, but what if it were applied simply to the application of transference conversation? So long as it allows lonely people to chat away without breaking the spell, that might be enough for GPT-3 to earn $3 a month.

Could there be a market for friendertainment?

https://phys.org/news/2018-12-americans-lonely.html

> Could there be a market for friendertainment?

Aren't there tons of English-speaking people in third-world countries capable of being worthy pen-pals and filling this market gap?

I actually just sent out an API request for a particular application that I believe GPT-3 should be capable.

I'm a university mathematics instructor. I find that a lot of students struggle with doing proofs and I think GPT-3 can help with that. A proof is essentially a sequence of logical inferences. I believe that, when given a proof written in natural language GPT-3 should be capable of detecting the logical implications made within. From there, it would be possible to correct student's mistakes or help them improve their answer.

Really? This seems like the exact sort of thing GPT-3 would be very bad at. The statements aren't the product of an internally consistent logical model, they are statistically plausible word sequences learned in a really clever way from a huge corpus of text.

That said, before dismissing it sight unseen there are effective and quick ways to test its powers with mathematics. You could prime an instance with the axioms defining a group, or a topology, and ask it to produce proofs of varying difficulty from intro abstract algebra or topology textbooks. Whatever the answer it would make an interesting blog post. What happens when you ask it to generate repeated outputs for the same prompt? What happens if you ask it to prove an untrue proposition?

So perhaps it could produce convincing looking proofs that are ultimately nonsensical.

It could generate math problems and have students try them out. Might be a bit of fun.

That's probably precisely an area you would NOT want to work with GPT-3. As Julian Togelius put it:

> GPT-3 often performs like a clever student who hasn't done their reading trying to bullshit their way through an exam. Some well-known facts, some half-truths, and some straight lies, strung together in what first looks like a smooth narrative.

https://twitter.com/togelius/status/1284131360857358337?s=20

Maybe in games where you want more convincing NPCs, something like Skyrim, where the main characters are very well branched, but it’s impossible to give every one depth.

This is one application of GPT that no one would mind ethically. Have a person oversee the logical/contextual structure.

I have a few ideas for disability aid applications.

~50-300 million sounds like an over-estimate, and the investment in GPT-3 is miniscule compared to SDC.

Plus, GPT-3 generates value even without direct product impact. I'm sure MSFT sales reps are already folding tons of nonsense about gpt-3 into their Azure pitches. Industry labs like OpenAI and DeepMind replace/augment the "Research & Development" model with a "Research & Marketing" model.

Seems spot-on. One trick to estimate a startup's burn rate is to multiply their number of employees by $200k. It's not too accurate, but it's within the ballpark.

So how many employees does OpenAI have? Supposing they have 500, that's a burn rate of $100M/yr.

250 employees, $50M/yr.

100 employees, $20M/yr.

I believe significant share of GPT-3 cost is machine-hours that were spent training this model - months of hundreds top-tier NVidia machines.

Edit: estimates range from $2-$5MM to $15MM

https://www.reddit.com/r/MachineLearning/comments/hwfjej/d_t...

I thought MS was giving them Azure GPU instances for free?

“Free”, which is basically Microsoft, as an investor, has paid those $4-$15MM minus ~30% of margin they make over pure operational costs.

Electricity was burned and NVidia GPUs that could have been doing paid job were doing GPT-3 training instead.

$15mm is a rounding error for MSFT

Gpt-3 is just one of openai's many projects. Theres ~20 authors on the paper, and they all almost certainly did not spend a full year on this project. So 300m is completely wrong.

Reply on news.ycombinator.com