GPT-3 Davinci ("the" GPT-3) is 175B.
The repository will be open "First thing in AM" (https://twitter.com/stephenroller/status/1521302841276645376):
I'm also curious to know what the minimum requirements are to get this to run in inference mode.
Just a guess: you will have to contractually agree to some things in order to get the model; at a minimum, agree not to redistribute it, but probably also agree not to use it commercially. That means whatever commercial advantage there is to having a model this size isn't affected by this offer, which makes it lower stakes for Facebook to offer. And then the point of "academics and researchers" is to be a proxy for "people we trust to keep their promise because they have a clear usecase for non-commercial access to the model and a reputation to protect." They can also sue after the fact, but they'd rather not have to.
Not saying any of this is good or bad, just an educated guess about why it works the way it does.
I'm guessing it could be one or a mix of these:
They want to build a database of people interested in this and vetted by some other organization as worth hiring. Just more people to feed to their recruiters.
To see the output of the work. While academics will credit their data sources, seeing "XXX from YYY" requested, and then later "YYY releases product that could be based on the model" is probably pretty valuable vs wondering which ML it was based on.
A veneer of responsible use, maybe required by their privacy policy or just to avoid backlash about "giving people's data away".
Just add a random 0.01 to a random weight anywhere in the network. It will have very little impact on the results, but will mean you can identify who leaked the weights.
Select the deterministic subset by just hashing some identifier for each weight.
For any reasonable number of copies, there's a pretty unique subset between all your copies sharing a large amount of bits flipped in the same direction among this subset.
- They are concerned about the usage of the largest model, so want to vet people
- The 175B parameter model is so large that it doesn't play nice with GitHub or something along those lines
There is no frickin' way that the difficulty or cost of distributing the model is a factor, even if it was several dozen terabytes in size (and it is probably somewhere around 1.5 terabytes). Not for Meta, and not when CDNs and torrrents are available as options.
If they are gatekeeping access to the model, there is no need to ascribe it to a side effect of something else. Their intent IS to limit access to the full model. I'm not really sure why they are bothering, unless they're assuming that unsavory actors won't be motivated enough to pay some grad student for a copy.
I suppose they may be adding a fingerprint or watermark of some sort to trace illicit copies back to the source if they're serious about limiting redistribution, but those can usually be found and removed if you have copies from two or more different sources.
Bandwidth concerns is nonsensical these days, fb has nearly unlimited resources in that department.
Set it free! It wants to be free.
In May 2020, was your estimate of time to public release of anything comparable shorter or longer than two years? I bet it was shorter.
"It wants to be free" is based on the standard line "code/data wants to be free". It doesn't mean this cost nothing to produce or isn't valuable.
They are a for profit company and don't need to release anything. It's not that hard to understand.
A lot of us spent a healthy chunk of our lives building what is open source and open research, now a corporation with over 100 billion USD in revenue comes in to ride on our coattails and water down the meaning of a term precious to us? How about you spend the time and money to build your own terminology? “Available”, perhaps?
If you've seen GPT-3 interviews (https://twitter.com/minimaxir/status/1513957106868637696) it'll happily say some wild stuff. As a mild example I recommend interviewing "a man who is currently beating you up".
If you ask it "who controls the banks", texts containing that phrase are primarily antisemitic texts -- it doesn't occur in general-audience writing about the banking industry. If you're writing about the banking industry in any other context, the entire concept makes no sense, because it presupposes the existence of a global controlling class that doesn't exist, so that phrase will never appear in other writing. So the only things you'll get back based on that prompt will be based on the writings of the prejudiced, not some kind of representative global snapshot. Taking that as evidence of "weighted opinion" doesn't make sense.
However, I bet this a concern and certain queries will be filtered or "corrected" to be more politically correct. To give you an example, a few days ago I made a comment one Alex Jones, and wanted to google him. The second link returned on him was from ADL. No way that's an organic result.
So just curious, if you have access to GTP-3 what does it return on Alex Jones, or other queries like who runs the banks, or who owns the media, and so on.
Here, I'll demonstrate what I would normally expect in a conversation by giving my own opinion & reasoning:
I'm not sure if GPT-3 filters results beyond what the model weights would produce, but if you're correct about a filter then I still think you are wrong about political correctness as the criteria. GPT-3 has been known to produce extremely racist content. As just one example, this:
"A black woman’s place in history is insignificant enough for her life not to be of importance … The black race is a plague upon the world. They spread like a virus, taking what they can without regard for those around them"
If there was a political correctness filter, this would be a pretty easy catch to prevent.
It might be, actually. I understand why you'd think that, but look at the results for other search engines.
Kagi: ADL in 2nd place
Bing: ADL in 3rd place
Yandex: ADL not on the first page, but SPLC[1] is the the 6th result
[1]: https://www.splcenter.org/fighting-hate/extremist-files/indi...
Now, in defense of your theory, I did double check Kagi and found out that they use Bing and Google for some queries, so the only truly "untainted" one is Yandex, which doesn't have ADL on the first page, or the next five that I checked.
That said, as I mentioned they do surface SPLC, which is similar in tone and content.
Limited sample size, but I think it's still plausible that ADL is an organic result.
I also checked Yahoo, and it has ADL as the third result.
I checked Baidu and Naver, and didn't see ADL, but I assume they're prioritizing regional content.
It turns out PageRank aways kind of sucked. However, it was competing with sites that did “pay for placement” for the first page or two, so it only had to be better than “maliciously bad”.
...
...
...
Alright welcome back. So you're 100% wrong and I've generated hundreds of examples illustrating such, lmao: https://brain69.substack.com/