The paper that made ChatGPT possible
arxiv.org
arxiv.org
That said, still mind blowing how far ahead Google was (is?) on research, data, compute, and how hard it is to actually novel things with that tech (vs OpenAI which... Just ships things).
Sam Altman practicing what he preached at YC for so many years. There's something wholly respectable about someone whose words closely match their actions.
And haven't we all been better for it?
Hardly "practice what you preach". They may have delivered technologically, but they have completely backtracked on their openess promises.
I ran stable diffusion at home. Never could do the same with anything out of "Open"AI
My comment was about Sam Altman doing what he was telling startups to do for many years at YC, ie - ship product and iterat
chatGPT you just have to interact with it and get a feel for its strengths and weakness.
"This paper presents the Transformer, a model architecture that relies entirely on an attention mechanism to draw global dependencies between input and output. Experiments on two machine translation tasks show that the Transformer is superior in quality while being more parallelizable and requiring significantly less time to train. On the WMT 2014 English-to-German translation task, the Transformer model achieved a BLEU score of 28.4, improving over the existing best results by over 2 BLEU. On the WMT 2014 English-to-French translation task, the Transformer model achieved a BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. The Transformer was also successfully applied to English constituency parsing both with large and limited training data. The Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution."
How accurate is the summary?
That's a very good question. I think GPT-3 shows knowledge is not in the brain, or in the neural network, but in the language itself. A human without language is not very smart.
The previous approaches, like LSTM, GRE, etc were also working well on sequence data, but were much less efficient.
Incidentally there's a link on the front page about this, https://news.ycombinator.com/item?id=34653462
"...please use the original title, unless it is misleading or linkbait; don't editorialize." [0]
I truly believe that the next Google will be founded this year (if it is not already here).
It seems like we lack a word in English for knowing too much about a subject that you can't see the forest for the trees.
To say chatGPT is not revolutionary is as good an example of not seeing the forest for the trees that I can think of.
What comes across is someone who downplays both the effort involved with and the importance of the “boring” stuff, the tweaks and product design that tie everything together, but we actually don’t know that he thinks like that. We just know that he said some things that can be excerpted to sound that way.
This would be like saying software isn't novel - everything is just an extension of that first NAND/NOR gate or that it's not useful because the computer doesn't actually understand the knowledge it's storing and is therefore no better than a clay tablet.