stellaathena

EleutherAI’s unofficial motto is “anything safe enough for Microsoft to sell for profit is safe enough to release.” You’re kidding yourself if you think Microsoft cares who they sell it to.

There are worthwhile conversations to have about democratizing technology, but this stuff is already out there regardless of what you or I do.

It’s a lot less expensive to hire a dozen teenagers to write fake news than use GPT-3. The cost to train GPT-3 is about the same as the cost to hire 1,000 people and pay them $10/hour to write responses for a year. Not to mention the fact that inference isn’t free.

“Works with Google” and “gets free compute from TFRC” are not really the same thing.

This is dead right and very important. The data you train on is much more important than model architecture in terms of validation and compliance, and yet it’s a closely held secret. Producing or obtaining good data is a pain in the ass.

For this reason, EAI has made the data we are training on public. I can’t link to it because of anon policies at conferences, but if you look at our website I’m sure you can find a paper detailing it and a link to download it.

Right, this is why a cloud company is sponsoring the replication. They can’t sell GPT-3 as a service until they get a copy of it.

Or for extra irony, ClosedAI

Hey! One of the lead devs here. A cloud computing company called CoreWeave is giving us the compute for free in exchange for us releasing it. We're currently at the ~10B scale and are working on understanding datacenter scale parallelized training better, but we expect to train the model on 300-500 V100s for 4-6 months.

... is it in the collection?

If I Google “bibliotik download” the first page of results has a link to download. Calling that “not easy to obtain” is absurd.

And yes, your last sentence is correct. That’s exactly the position being taken.

TFRC, a program that lets you borrow Google's TPUs when they're not being used. You can apply here: https://www.tensorflow.org/tfrc

I'm not sure what a BDFL figure is, but Google does not give us millions of dollars. We are a part of TFRC, a program where researchers and non-profits can borrow TPUs when they're not being used. You could say that we are indirectly funded as a result, but it's nowhere near millions of dollars and it doesn't reflect any kind of special relationship with Google.

There's more than 800GB of fanfiction in the world.

Not currently. The code is on GitHub, but we do not have a public-facing model. We felt that the world doesn't need another GPT-2 replica, but if there's interest we can look into doing so though. We are planning on making our GPT-3 replica public facing though.

https://github.com/EleutherAI/gpt-neo

To be more specific about what the other responses say:

Being "equivalent to a TM" and being "equivalent to the Python compiler" mean the exact same thing. Pretty much every widely used model of computation is equivalent in computing power to Turing machines. The notable exceptions are all weaker than Turing machines, such as arithmetic circuits.

For those who don't play Magic, this is on a scale where tier 1 decks are top decks, tier 2 decks are reasonable, tier 3 decks have serious weaknesses but are playable.

This seems like a bizarre rule to add. Why is this result even a bad thing for Magic?

This is not correct: the paper does have to do with optimal strategies. They even say as much in the abstract.

>In this paper we show that optimal play in real-world Magic is at least as hard as the Halting Problem, solving a problem that has been open for a decade

Lots of things are more complicated than a computer can handle. There's an entire hierarchy of non-computable problems by complexity (actually several).

Did you actually read the paper? There's a mathematical definition of complexity that is used, and on that metric it is shown to be far harder than any other previously analyzed game.