Hacker News Reader

Comment by naillo | Hacker News Reader

This is really cool but the output is such garbage at that weight size that you might as well be running a markov chain.

View on HN

That's why Alpaca is so exciting: it instruction-tunes LLaMA to the point that even the tiny 7B model (the one that fits on a phone) produces useful output: https://simonwillison.net/2023/Mar/13/alpaca/

I’ve been playing with the Alpaca demo, and I’m really impressed! The outputs are generally excellent, especially for a model of that size, fine tuned on a $100 (!!) compute budget.

If the cloud of uncertainty around commercial use of derivative weights from LLaMA can be resolved, I think this could be the answer for a lot of domain-specific generative language needs. A model you can fine tune on your own data, and which you host and control, rather than depending on a cloud service not to arbitrarily up prices/close your account/apply unhelpful filters to the output/etc.

But they won’t give us the model… so it’s ultimately meaningless because they’ll just sell out

My understanding is they legally can't. It was trained used OpenAI, which doesn't allow using their output to train new models. Someone would need to find another data source to fine tune llama.

You don't need to find a new data source, you just need to find an unencumbered third party. You can use the that data publicly provided in the git repo as long as you haven't signed an agreement with OpenAI yourself.

Why they can train their model on copyrighted data and claiming fair use as they do not outright copy while disallowing training other models on their output? I understand revoking access though.

They realistically can not, I believe. If Microsoft is right that this is under fair use, then you are only limited by OpenAI ToS, which also says that the copyright of generated content belongs to the user. Do the following:

Person A uses GPT3 to generate training data, and publish it on his blog, without representing it as human generated. Person A does not give permission for it to be used by Alpaca team.

Alpaca team comes along, scrape his blog, and uses it as training data, without permission from person A. Now this is fair use, so there is nothing person A can do to stop it, just like how Github scraped our code for Copilot without permission.

It’s only a matter of time

waiting for the stable diffusion version of gpt still

What will OpenAI do, sue? Okay but now it's out there.

I believe the work was done by Stanford, so OpenAI could revoke Stanford's access to their API. That would inhibit Stanford's ability to do new research with this system.

Also how could that even be under protection. As if they haven’t been scraping copyrighted materials and sites with end user agreements to train the model in the first place

OpenAI I knows they’re just a few cycles behind open source models - probably why they stuck a deal early with Microsoft.

If they don't release the model, recreating it doesn't look too hard. $100 worth of compute time to run the fine-tuning, and the training data they used is here: https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpac...

That would have the same licensing problems that they have though: that alpaca_data.json file was created using GPT3. But creating a "clean" training set of 52,000 examples doesn't feel impossible to me for the right group.

You're only bound by the terms of OpenAI's agreement if you agreed to the terms of use. If a third party obtained the data without signing an agreement with OpenAI (eg. by just downloading it from that repo) they are under no obligation to refrain from using it to compete with OpenAI. It is fair-use by the same argument OpenAI itself uses to train its own models on publicly available data.

How one can tune the model to a specific usage? Is there some place that teaches this?

It’s quite a bit bigger than GPT-2 which was a really big deal not very long ago (remember the unicorn news article example and the slow release because it was apparently too powerful?)

Our standards have gone up so much!

If you are talking about the video that's perfectly fluent English. There are some unusual elements to the story which probably wouldn't be there in a larger model.

I'd invite you to try that with a Markov model or even something like a LSTM based neural network and compare.

Afaik, the Llama sampler needs to be tuned to get more sensible outputs.

https://twitter.com/theshawwn/status/1632569215348531201

Isn't any LLM mathematically a Markov chain, such that the current state includes the context of the last (finite) n tokens?

The distinctive aspect of the 'transformer' family of neural networks is that they incorporate 'attention', which is a model that identifies which parts of the input are critical to its meaning. You're correct in that modern transformer models are essentially Markov chains, but the function that derives the probabilities (of which the attention heads are part of) is imbued with 'understanding' of related concepts during the training process. In contrast, a traditional Markov chain text generator might have a probability function that only takes into account the frequency of the n-grams in the training data, and so produces superficially coherent (but mostly meaningless) output.

This is like saying all computers are state machines because they have finite amounts of memory and disk space. It's sort of true, but it's not a useful mathematical model.

With a Markov chain, you're assuming a state machine where each state has independent probabilities on outgoing edges. As the number of states gets larger, you have fewer training samples for each state. When n gets large enough, nearly all states have zero training samples; they've never been seen before. How do you estimate probabilities?

Better to just say it's a stateless function of the input.

From the video output seems fine.

But if it is a trimmed version, it is wong to call it LLaMa.

Could call it Slim LLaMa

SLLaMa?

It's nonsensical, celeb announces they're going to rehab and notes it (?) is an issue affecting all women, at least, earlier today (??), they also noted it wasn't drugs or alcohol this time, but, a life (???)

Without instruction tuning, the perfect language model produces output which has the same level of intelligibility as random text from the training set. And the training set probably has a lot of spam and junk in.

What are you comparing it to? Without instruction tuning and a two character prompt "He" I am not sure why you would expect it to perform any better.

I was replying to a comment that said it “seems fine.”

It does not seem fine.

It is incomprehensible and doesn’t match the results I’ve seen from 7B through 65B.

It is true that RLHF could improve it, and perhaps then this severe of optimization will seem fine.

I've heard a number of people say (from earlier) that the quantization and default sampling parameters is way wacked. Honestly even running that model size alone is the big achievement here and getting the accuracy to actually reach the benchmark is the beeg next step nao, I believe. <3 :'))))

If you run a quantized 60G model and the output is worse than raw 7G model, you can throw your quantizer out.

Reply on news.ycombinator.com