Microsoft says GPT 3.5 has 20B parameters? | Hacker News Reader

Microsoft says GPT 3.5 has 20B parameters?

arxiv.org

57 pointsHeidaradar25 comments

It's against HN's guidelines to editorialize titles like this. From https://news.ycombinator.com/newsguidelines.html:

"Please use the original title, unless it is misleading or linkbait; don't editorialize."

If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...

This seems like a valid exception from this rule. The interesting tidbit here is what the title suggests, even though it is not the title of the paper. It should be possible to point out to facts like these hidden in papers, reports, etc without having to create a separate article about it. You can of course add a comment as you suggested, but that does not reflect how people use HN.

I just want the 100k context window Anthropic gave Claude, but for GPT. Claude likes to hallucinate when I ask it to build chapter notes for a class I’m taking, and I don’t want to have to break up the text into tiny bits for GPT…

The large context is the reason its hallucinating and gpt would likely have the same problem. You can see this effect even on gpt-3.5-turbo-16k. Performance and accuracy are directly correlated with context size

That's just GPT3.5 having a pretty spotty attention. This becomes much less of a problem if you use gpt-4-32k.

*confabulate. Hallucination is StableDiffusion

The lost-in-the-middle phenomenon makes me _not_ want larger context windows because the model cannot effectively use it. It seems like we need another arch to process large ctx size.

Performance on long contexts just isn't very good right now.

Relevant paper: Lost in the Middle: How Language Models Use Long Contexts https://arxiv.org/abs/2307.03172

Link to the paper directly https://arxiv.org/pdf/2310.17680.pdf

The system that they describe, Codefusion is interesting because it's a diffusion model for generating code rather than an autoregressive model like most LLM code generators.

Also on HuggingFace: https://huggingface.co/papers/2310.17680

> We evaluate CodeFusion on the task of natural language to code generation for Bash, Python, and Microsoft Excel conditional formatting (CF) rules.

Microsoft Excel conditional formatting, really? The future we want is one in which that abomination has died out, not one in which it is thriving.

ding ding ding ding !! prizes for the spectator who correctly names the Tip of the Iceberg for AI-assisted-communications at every level. You have heard of evolution? creativity ? Free-Will ? now, with AI assisted communication technology, every road leads back to MSFT ! So you wondered about Proust, or that line in Hamlet? surprise - it is only the entry topic to get back to a clever rhyming ad-jingle for your next software subscription update. btw- our agent is authorized to check your bank account balances while you renew. according to that user agreement you acknowledged.

/fiction

I know you're being tongue in cheek but this is why the evolution of open source models capable of cheaply running on your own hardware (i.e. the r/locallamma folks) is so important.

It says gpt3.5-turbo has 20B parameters, which I believe. But it says gpt3.5 (text-davinci-003) has 175B parameters, which I also believe.

This was expected because open source models of the same size already beat GPT-3.5 in many ways. And Mistral 7B makes you think if huge parameters are even needed for something like GPT-3.5 level.

Please enlighten me. Which open-source models beat GPT3.5?

I could be wrong but I believe the original comment that's used as the title here comes from this tweet: https://twitter.com/felix_red_panda/status/17189166315129492...

Didn't it leak earlier that it is 100 billion? And GPT 4 is 1.17 Trillion?

That 1 trillion number was quickly confirmed to be false.

It's believed that GPT4 is a Mixture of Experts model -- it's 8 smaller models (estimated around 200B each) and a router picks the best model to redirect to based on input.

So 800B + a router network, so not far off a trillion?

That sounds incredible given how powerful the model is.

Mhm, interesting development, the paper has been withdrawn.

That would certainly explain the pricing (gpt-3.5 vs davinci).

So it's possible to run gpt-3.5-turbo on a local machine?

Did openAi ever publicize how many params GPT-4 had?

Just eating up water and fossil fuels.

Reply on news.ycombinator.com