Hot new startup just dropped:MSFT
656 karma · joined June 11, 2020
Not that I think either are competent
Sam tries to sound smart while not really having any technical insight. He does a tremendous job with it though.
One way to think about this is: at some point in the next few years we'll have a few hundred GPUs/TPUs that can provide the compute the compute used to train GPT3.
This discovery was always going to happen. The question is if OpenAI made radical scaling possible unlike before. Answer there is also a no. There are clear limits to number of collocated GPUs, nVidia release cycles, TSMC capacity, power generation etc.,
So in the best case OpenAI fudged the timeline a little bit. Real credit belongs to the Deep Learning community as a whole.
The goal of demonetisation has never been clear. It was a thuglaq move and Indian right wing has retro fitted motivations to it.
The UPI revolution happened despite Modi, not because of him in the slightest.
It is less apparent now but still, the repeated flex of “Google”, 20% project etc when no typical reader would assume them is classic corporate charlatanry.
Shame because I like the project otherwise
The OP doesn’t explain anything. It just vaguely talks about a few things that might break when scaling context. But that means nothing.
Take for example sinusoidal embeddings they talk about. Of course it breaks for large contexts but in no one uses it. GPT uses learned positional embeddings so the entire section is irrelevant.
Copy this for pretty much everything else.
Being an expert in a field has never been this exhausting
There are more mosquitoes than tigers
You’re speaking on an agenda without accounting for reality