Roll out your own ChatGPT | Hacker News Reader

Roll out your own ChatGPT

harishgarg.com

38 pointshgarg23 comments

... using the GPT-3 api.

Not something you can run on your own machine (or data center) unfortunately.

You can run the OpenAI completions endpoint with the text-davinci-003 model.

Is that model opensource or downloadable?

You can run GPT-J and GPT-Neo on (2) 24Gb GPUs, like a pair of 3090s. Not to mention Stable Diffusion which has much lower requirements to produce amazing AI images. Now, if you want to train a huge model from scratch that’s a different story, but running your own server is definitely possible and improving every day.

How much would 2 24gb GPUs cost? Unfortunately I think it’s still unaffordable for most

Maybe you could rent GPU space on a service like vast.ai. But I imagine it would be difficult to set up, and at that point currently you’re better off just using GPT3…

GPT-J specifically was written using Google Cloud's TPU stuff.

https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b

They even have a google colab notebook thingie. Not much setup needed, just an account.

https://colab.research.google.com/github/kingoflolz/mesh-tra...

Using the actual title might fend off some of the criticism.

"Building a chat app with GPT-3, ReactJS, and NextJS..."

Not

"Roll out your own ChatGPT"

which is rather unrealistic.

On first skim, it looks like a useful guide so good luck with it.

Shouldn’t this title be - build a webapp using ChatGPT API?

Yeah that was disapointing given the title. ChatGPT didn't spring full-formed out of nowhere. It's progress upon an existing body of work that is totally opaque to the uninitiated but a reasonable facsimile can be approximated out if you spend enough time on the right discords and http://huggingfaces.co and Google Colab. ML algos and their results shouldn't be limited to an Ivory tower of Arvix papers.

No, because there is no (public) chatGPT api. This is using the GPT-3 API to build something that behaves similarly to chatGPT.

I wonder if one of the open models would work the same

I tried them all, and continue trying when new things come out. No open model is capable of even remotely approaching the capabilities of GPT-3 or ChatGPT. Even if you had an equivalent trained model, running it would require enormous hardware.

While Stable Diffusion is pretty comparable to DALL-E 2 even on small hardware (such as M1 MacBook Air), this doesn't hold true for large language models. These need much, much more VRAM.

text-davinci-003 is extremely close, or code-davinci-002

None of that model available to run on your hardware

thats a real shame. Any idea why GPT-3 generations differ so much? Maybe eleuther/whatever could look at refining their model in a similar way.

It's most likely because of the actual runtime size of the model. All the open models are sized for consumer-grade devices, and thus 10x-100x smaller than whatever OpenAI runs (probably around 100+ GB in VRAM, maybe even some multiples more). This is one of the main reasons why their API makes business sense - it's not practical at all to run models like GPT-3 yourself, and training them costs incredible amount of money too.

FYI ChatGPT does not have an API and the article references GPT-3 general availability release.

ChatGPT runs on a heavily modified version of GPT3 nicknamed “GPT3.5” that is also not available via API.

This can be a good reference implemntation for those starting out as a hobby project.

Is there a working demo, video, screenshots, or git repo for this solution that gives us a glimpse of what to expect once this is built?

FYI I am planning to add a straightforward way to deploy dialog agents backed by OpenAIs new model text-davinci-003 to my site aidev.codes over the new few days if I have time.

So you will be able to mainly focus on the prompt template and the rest will be handled including the hosting, probably by something like a template app built in the system I have now. The main thing missing to enable easy creation of dialog agents is for me to add a completions endpoint and maybe things like socket.io integration in the template app.

I had a problem running the code see

https://stackoverflow.com/questions/74924661/typeerror-opena...

I'm 90% sure this was written by / with assistance from ChatGPT; I previously prompted it for some React examples and some of this structure of how to setup the env is extremely familiar.

does gpt-3 keep track of a conversation though?

Is the API nerfed like the live chatbot is?

Nope, the nerfing comes from the RLHF training to make it fit the character they wanted.

Reply on news.ycombinator.com