Low-code, no-model recommendation system with GPT3 | Hacker News Reader

Low-code, no-model recommendation system with GPT3

medium.com

95 pointssatorii62 comments

I wish there was a light version of gpt3 with maybe like 100 api query limit per day so just amateurs like me got to play with this API too on weekends.

I have no agenda of changing the world and I have nothing substantial to fill in the gpt3 invite form but sometimes the best ideas come from tinkering and I just wish they gave some limited access to everyone.

You might like: https://transformer.huggingface.co The GPT-2 model is very similar to GPT-3 just smaller.

Thanks for the link. So as someone who is completely new to this, if this guy used this model instead of gpt 3 would he get similar results?

No, because the model is smaller.

out of the box probably not. gpt-3 is pretrained on a much larger corpus and is a much higher capacity model, which is why its been performing very well in zero-shot learning examples like the one above. That being said you should try it.

Note the huggingface library is a python library, so you'd technically be writing some code, though the library API is super user friendly and the community is very helpful.

If you want to play with your own GPT-2 API, I put this together and shared it on HN yesterday: https://github.com/cortexlabs/cortex/tree/master/examples/py...

You can run that and deploy your own GPT-2 API locally, or in the cloud. I'd also recommend checking out Max Woolf's gpt-2-simple, which is a library that makes it really easy to fine tune GPT-2 with your own text (if you've played the free version of AI Dungeon, this is how they trained their model):

https://github.com/minimaxir/gpt-2-simple

I thought they have a free tier from October? https://twitter.com/hardmaru/status/1301362995356774401

Not sure what the 100k tokens translates to in terms of query limit

That's 3 month max though

would be nice if it was longer, but should have decent ideas in 3 months. And having another go with a throwaway email probably not that hard

assume each prediction use 1000 tokens (including both input prompt and output text), then 100k tokens -> 100 predictions

"Recommendation system is so successful in many products and services we interact everyday" ->

"Recommendation system[S] [ARE] so successful in many products and services we interact [WITH] every[ ]day"

I'm sorry, but four errors within the first sentence is just not good enough - did the author even read this themselves just once before hitting publish? Show just a little bit more respect toward the people who give their attention to what you've written.

Maybe the author's first language is not English?

Sure, neither is mine. But if you don't have the skills to proofread what you've written yourself, then get someone to help that has. I don't expect to be able to write something in Chinese deserving of a wide audience all on my own either.

A lot of people write content like this for nothing or next to nothing. If the bar to publish a blog post is "needs an editor" then there's a lot of content that simply won't exist any more, because people don't want to pay money to read blogs (at least I know I won't).

Personally, I don't care about form if I find the content interesting.

Form is content, content form. How you do anything is how you do everything. Errors in grammar and syntax indicate errors in thought and care.

Not really, but the way you put it sure sounds nice.

That is quite probably untrue, and I'd wager many best selling and critically acclaimed books contained a number of 'howlers' on the publication of their first editions.

Thanks for pointing out the errors. I got them fixed in the blog. This is also a good use case of GPT3, checking grammar and spelling. I might build an application for this.

Thanks for fixing them - and sorry if I was overly harsh. Sharing your writing with the world is nerve-wracking enough already without having the spelling and grammar shot down on HN. Truth be told, my first thought was "maybe this is some example of text created by GPT-3?" But I think you're right that it would actually serve well as a combined spelling and error checker to catch things like that - I hope you do build an application for it, that would be neat.

Given that "Open" AI API is very pricey, Im waiting to see how economical is it to use this as it is heavily marketed by all these blog posts.

For anyone else curious, the pricing according to gwern [1]:

    Explore: Free tier: 100K [BPE] tokens, Or, 3-month trial, Whichever comes first
    Create: $100/mo, 2M tokens/mo, 8 cents per additional 1k tokens
    Build: $400/mo, 10M tokens/mo, 6 cents per additional, 1k tokens
    Scale: Contact Us

Expensive but (unfortunately) pretty standard market price for cloud NLP stuff. The price is about $40 per million tokens.

For comparison, cloud translation costs per million characters range from $6 (Azure bulk pricing) to $20 (Google/AWS without bulk discounts). If we assume the average BPE token length is 5 characters, GPT-3 will cost about $8 per million characters.

[1] https://old.reddit.com/r/GPT3/comments/ikorgs/oa_api_prelimi...

In one of the Reddit comments about a service (PhilosopherAI.com) using the API outlining the possible total costs:

> That makes for 400 million tokens in 2 or 3 weeks, which puts me at like $4000/mo minimum

Oh dear, that means...

> Scale: Contact Us

So while they are using the API for free for now, by these metrics, they are least going to be soon having a running cost of at least $4K+/mo. That is an expensive toy. The same goes for AI Dungeon.

> App Store earnings will be used to cover OpenAI costs & develop new features. Thank you for your support.

Well I will be seeing a tsunami of complaints from OpenAI projects that they can't pay up for their "game changers" since they will be priced out quickly. OpenAI Wins, Everyone else loses.

If this is for simple projects, then I wouldn't dare build an entire startup based on someone else's API. OpenAI will win by default here.

I was thinking about Ai dungeon the other day. I can't imagine it really is sustainable with those 10$ subscriptions covering the free play folks too. What happens whenever they get featured on some bigger website?

AI Dungeon free players only use GPT2 instead of the GPT3 API.

AI Dungeon has already scaled up to over 1,000,000 players and managed costs. They have a whole article about it here:

https://medium.com/@aidungeon/how-we-scaled-ai-dungeon-2-to-...

An important note, as others have said, is that not every player gets the same model. Free players, as far as I'm aware, only get GPT-2 ("only" is a strong word, as up until GPT-3, GPT-2 was the state of the art in text generation and is still extremely impressive in its own right).

What is a token? One character? Or is it a currency?

A character, both input and output.

Not exactly, GPT-3 uses a variant of BPE [1], so one token can correspond to a character, an entire word or more, or anything in between. The paper [2] says a token corresponds to 0.7 words on average.

[1] https://en.wikipedia.org/wiki/Byte_pair_encoding

[2] https://arxiv.org/abs/2005.14165, page 24

GPT3 really is remarkable.

If you told me in 2015 that I can give just a few examples of movie recommendations, on a model trained for general text, and get perfectly coherent recommendations, I wouldn’t have believed you.

Yes right, but just hope the signal to noise ratio in the trainings remains stable. Garbage in ...

Indeed. But now OpenAI has become more like another DeepMind, but starting to rent out their APIs to its users, indicating that they probably won't ever release models for their research.

They might as well rename to Standard AI.

I don't fully disagree, but I also think there is a layer of complication here that goes beyond OpenAI's hopes of monetization (which are clearly a factor).

I maintain an open source ML deployment platform, and I've interacted with a bunch of teams that have used it to deploy GPT-2. It was actually the platform AI Dungeon built their app on. GPT-2 is a beast to deploy—it's huge (almost 6 GB fully trained), requires GPUs, and scales fairly poorly. You need to autoscale GPU instances aggressively to handle any kind of real time inference situation with it, and even with spot instances, that gets expensive quick.

GPT-2 is 1.5 billion parameters, and at the time, was scandalously large. GPT-3 is 175 billion. For a model that large, there's real questions around whether it's even feasible for the average team to use it if it is not hosted somewhere else as a third party API.

From that perspective, I think the value OpenAI captures with the API is less about the exclusivity of the model itself, but the exclusivity of their infrastructure. Because of that, I wouldn't be surprised to see them open source the model for research.

However, I 100% agree that the fact that the model still isn't open is concerning, and it casts some doubts on whether or not it will ultimately happen in the future.

I'm not so surprised given that the paper generator has been around for much longer:

https://pdos.csail.mit.edu/archive/scigen/

Example:

https://pdos.csail.mit.edu/archive/scigen/rooter.pdf

In a way, I find the output better than GPT-3's. GPT-3 output is always a bit creepy to read, as if you are talking to a deranged person.

Nice as a stunt, but feels like taking a 40 ton truck to get your lunch sandwich from the bakers.

GPT-3 is the Rube Goldberg of AI solutions

I guess it depends how you think about it. From computational perspective, it's heavyweight. On the other hand, it really optimizes for minimal human effort, on several levels:

* GPT-3 is conceptually simple: the effort to invent it and engineer it amortizes well.

* Once the model is trained, it can be reused to very different tasks.

* Querying doesn't require programming knowledge.

If this turns out to be easy and convenient, it won't matter whether the tech is overpowered.

A family car is really overpowered for most trips to the bakers.

Yet people take the car all the time, instead of walking or cycling, because it's just more convenient and easy, and people are lazy.

You can't write just this off as a stunt.

If you described how YouTube renders cat videos to Tim Berners Lee in the mid 90s, it would also seem like a 40-ton truck.

And yet here we are.

Serendipity is another GPT-3 based recommendation engine. Check its about page. http://serendipityrecs.com/

Hey, cool, you built the same thing as me but with less code: http://serendipityrecs.com/

One thing that's cool about GPT-3 is how versatile it can be, so I'm exploring a natural language search interface, which is something that traditional recommendation algorithms can't do.

Are you planning to incorporate GPT-3 into your startup? I'd love to hear about what use cases you envision it solving. If you're interested in connecting, my email is in my profile and my website.

Cool! I will drop you a message or catch up with you on openai-api slack.

There are a couple of features we see GPT3 stands out and potentially benefits the product features.

Currently we are evaluating the end to end quality and cost versus our traditional ML based recommendation and semantic search pipeline.

Well it's a no-model AI system if the interface is a HTTP endpoint that's out of your control...

Very creative use of the technology. I wonder when GPT3 will evolve to handle more languages, and what if any steps they take to reduce cultural bias? If I was in Japan these recommendations would certainly be less useful. :)

Given this blog post I have no idea if it even works well for English. Sure, the demo video contains sensible recommendations but what gives? Most of those I've seen seem to be included in the training data shown in the image below. If you have the term "AI" in your blog's blurb readers might expect the author to actually perform an evaluation of stuff they write ads about.

Presumably when a new movie (not included in GPT3 training data) comes out the recommendations are not going to be too good, unless someone spends a lot of money retraining the underlying model?

This will be the case for everything built on GPT-3, but you've got to imagine it's possible for them to keep refreshing it, eventually training or fine tuning it on a daily basis as their web crawl updates.

I am curious how much it actually thinks and how much it simply uses text from recommendation engines that got crawled.

That is a very deep question.

A neural network can't think, please stop cargo culting neural networks

Panpsychism is a serious philosophical position.

Could you please give an explanation of panpsychism applied to software that make sense? Spoiler: you cannot and it's totally refuted empirically. If I code a hello world, what kind of cognitive abilities does this have?

BTW I guess panpsychism researchers should start talking to rocks maybe one day they'll find one that is an AGI! /s

The position is that information not physical matter is ontologically primary and that substrate independent integrated information processing is identical to consciousness.

A hello world program and a rock are not integrated information processing systems. The former is an automata which can be contained in an integrated information processing systems and the latter is a more arbitrary construct which does very little information processing at all (much less integrated).

Sounds like the ether in physics around 1900 or the four elements in Greek philosophy.

Non-falsifiable dreams.

Suppose there’s a highly-detailed molecular dynamics simulation of every atom in your brain: every ion channel in every neuron, any neurotransmitters wafting about, and whatever other mechanisms might be relevant (ephaptic coupling?) for good measure.

This would be frightfully expensive to build and run (BlueBrain blew billions on a very stripped down version)l Nevertheless, it’s not obvious to me that this MD model wouldn’t “think” in the same way you do.

The simulation you describe, if precise enough is isomorphic to a real brain so obviously it's tautologic that it would think the same way as the real brain do (qualias aside).

But panpsychism apply to anything, e.g a rock. And your statement does not apply to a standard neural network.

Still seems like an interesting question if you replace “think” with “inferred”, “learnt” (in the statistical sense), or any other synonym that’s in opposition to “memorized and parroted”.

Slightly related excerpt:

"Supporters of the Strong AI Hypothesis insisted that consciousness was a property of certain algorithms – a result of information being processed in certain ways, regardless of what machine, or organ, was used to perform the task. A computer model which manipulated data about itself and its ‘surroundings’ in essentially the same way as an organic brain would have to possess essentially the same mental states. ‘Simulated consciousness’ was as oxymoronic as ‘simulated addition’.

Opponents replied that when you modelled a hurricane, nobody got wet. When you modelled a fusion power plant, no energy was produced. When you modelled digestion and metabolism, no nutrients were consumed – no real digestion took place. So, when you modelled the human brain, why should you expect real thought to occur?"

— Permutation City by Greg Egan

Intersteller (with -er, indeed)? How did it pick that one up as an alternative for Inception? There is something with that title, but no information in IMDb.

Anyway, for the rest it looks similar to Netflix' recommendation: similarity in actors, directors, keywords. A toy.

A2:B10 (including the results for Intersteller{sic}) are the prompts, it's very confusing in the video. The results for "The Irishman" are from GPT-3.

No model? What? Of course this is a model, it's just a pre-trained one.

Reply on news.ycombinator.com