Hacker News Reader

Comment by eigenvalue | Hacker News Reader

I'm not surprised-- I recently suggested that someone might try to pull an Aaron Swartz with the LLAMA weights (i.e., release them in an uncontrolled way similar to how Aaron attempted to release the JSTOR database). It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics. If the paper is to be believed, this is a major development, allowing you to get close to GPT3 performance on a single GPU (at least for inference on the smallest model). Clearly some renegade academic feels the same way.

View on HN

> It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics.

I mean at least they didn’t pick a name that heavily implied they were, are, and always will be open. Then do the opposite. You know, like OpenAI?

So now we got some weights I guess.

> It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics.

Facebook almost certainly knew leaks would happen. My guess is keeping the model "contained" was a legal shield more than anything else, to protect themselves from liability in the case someone misuses the model.

It was already the most open language model in its class, given that the code for training and inference was available and it only used public data for training.

For Google and OpenAIs offerings, have fun reimplementing it from descriptions in the paper (including small crucial details that they may have left out), training it for a month, and then wondering if the implementation or the training data is the reason your model isn't as good as theirs.

The training code is not available.

Weights are more valuable than training code in one regard. Even with the training code you may not have the dataset and reproduction requires a massive GPU cluster that few can afford.

Weights are more valuable to random individuals who want to mess around with the model. Training code is more valuable to other companies that have the resources to use them, because then they can tweak/modify however they want. But even then, you still need the training data, which in the case of OpenAI and DeepMind is a big part of the secret sauce (not just the raw data but also the process for cleansing and de-duplicating it).

Training code won't get you much if you don't have the infra/money to gather a suitable dataset or actually execute training. Plus if your goal is to "steal" or riff on the base model, it's already there in the weights. Also probably not difficult to figure out how to fine tune it once you have the weights and tokenizer.

>Clearly some renegade academic feels the same way.

Or someone pretending to be a renegade academic. It's not like there is a KYC process.

Probably done with meta's blessing. Pretty obvious the scientists wanted to release it and legal stopped them.

"It's quite misleading for FB to claim that they are being so open, but then hoard the weights and only release it to a few academics."

I don't think it's misleading. Even saying that Facebook "hoards" the weights when they are more open than any other major AI company is baffling.

it's really easy to get access to the weights, I got access to them as a master's student with just a single very minor publication in the domain, and basically all my colleagues who applied did too.

there is no Aaron Schwartz thing here, a huge amount of of people will have them, someone was going to leak for sure

if aaron had attempted to release the jstor database he would have succeeded

Didn't he get caught before he fully downloaded everything? My understanding was that's what he was doing with that laptop stuffed in a supply closet on the MIT campus.

probably nobody knows

but certainly he was capable of ensuring that everything he downloaded would be released if that was what he wanted

You should add your Matrix to your contact page.

Edit: Also, is there going to be another der* release this year? They and therefore you have taught me more than I'd learned in the last five years otherwise.

He could have been trying to evaluate how many public domain papers it wasn't sharing access to at the time, which would have required a collection of the database. Would also have explained why he wasn't trying to hide his behavior.

He would have succeeded if he was just wanting to deaddrop all the data in a bin, probably without getting caught.

Reply on news.ycombinator.com