Hacker News Reader

Comment by famouswaffles | Hacker News Reader

>They are not machines that reason, they are approximators.

And meaningless distinction of the year award goes to..

"It's not real [insert property]" is not an intelligent argument. By all means, divine the way to distinguish results of the two. After all, what kind of important distinction can't be distinguished by results?

>Further it didn’t happen over night but through successive improvements

The only difference between GPT-3 and GPT-2 was scale. They didn't even change the tokenizer until 4. There were no "successful improvements" to smoothen the massive gap in capabilities between the two. So to say that was expected just shows more lack of knowledge here.

View on HN

> The only difference between GPT-3 and GPT-2 was scale. They didn't even change the tokenizer until 4. There were no "successful improvements" to smoothen the massive gap in capabilities between the two.

"Other than that, how was the play, Mrs. Lincoln?"

GPT-2 is two orders of magnitude smaller. In terms of forebrain neuron count this is the difference between a human and an elephant shrew or budgerigar. It was absolutely expected by reasonable people that 2 OOMs of scaling will provide a qualitative jump.

>GPT-2 is two orders of magnitude smaller. In terms of forebrain neuron count this is the difference between a human and an elephant shrew or budgerigar. It was absolutely expected by reasonable people that 2 OOMs of scaling will provide a qualitative jump.

When GPT-3 was released, it was by far the largest artificial neural network ever trained. And I mean by far. Now there wasn't any big jump in hardware capabilities to spur this sort of gulf. It wasn't a case of "Oh now we can train a very large model"

So Want to know why there was such a gap ? It's because most researchers assumed the models would overfit the data or display diminishing returns long before 175b.

Brain neurons are not comparable to ann parameters.

You would expect a machine that can reason to improve in its responses as a conversation progresses, instead of hallucinating.

Reply on news.ycombinator.com