Hacker News Reader

Comment by Roritharr | Hacker News Reader

I have the 65B Model running on my MacBook here. Honestly, I wish I knew enough about how to get it to output something useful. It's like poking a drunk librarian.

View on HN

Le sigh. Do I need to fix this sampler too? The default output should be excellent, not a drunk librarian.

Rip open the code and make sure it has top_k 40, temperature 0.7, no top_p, and a repetition penalty of 1/0.85.

Proof: https://twitter.com/theshawwn/status/1632569215348531201?s=6...

Implementation: https://github.com/shawwn/llama

It feels basically identical to davinci. I think no one realizes it yet because Facebook released a horrible sampler that everyone copied.

I can set top_k, temperature and top_p via shell already, but I don't think repetition penalty is dynamic yet. Looking into it.

Edit: This fork ( https://github.com/beiller/llama.cpp/tree/feature/repeat_pen... ) adds repetition penalty via two values, repeat_last_n = 64, repeat_penalty = 1.300000.

Playing with those now.

I wish I could somehow force it to output longer texts. It just stops at arbitrary points, often in the middle of sentences.

Thanks for flagging this. Using that repo, I'm getting pretty good results with the following:

./main -m ./models/13B/ggml-model-q4_0.bin -t 7 -b 10 --top_p 0.0 -n 5000 --temp 0.7 --repeat_last_n 64 --repeat_penalty 1.3 --prompt "You are a question answering bot that is able to answer questions about the world. You are extremely smart, knowledgeable, capable, and helpful. You always give complete, accurate, and very detailed responses to questions, and never stop a response in mid-sentence or mid-thought. You answer questions in the following format:

Question: What’s the history of bullfighting in Spain?

Answer: Bullfighting, also known as "tauromachia," has a long and storied history in Spain, with roots that can be traced back to ancient civilizations. The sport is believed to have originated in 7th-century BCE Iberian Peninsula as a form of animal worship, and it evolved over time to become a sport and form of entertainment. Bullfighting as it is known today became popular in Spain in the 17th and 18th centuries. During this time, the sport was heavily influenced by the traditions of medieval jousts and was performed by nobles and other members of the upper classes. Over time, bullfighting became more democratized and was performed by people from all walks of life. Bullfighting reached the height of its popularity in the 19th and early 20th centuries and was considered a national symbol of Spain. However, in recent decades, bullfighting has faced increasing opposition from animal rights activists, and its popularity has declined. Some regions of Spain have banned bullfighting, while others continue to hold bullfights as a cherished tradition. Despite its declining popularity, bullfighting remains an important part of Spanish culture and history, and it continues to be performed in many parts of the country to this day.

Now complete the following questions:

Question: What happened to the field of cybernetics in the 1970s?

Answer: "

Interesting, that one segfaults for me. Gonna try with fresh eyes tomorrow, thanks!

> and never stop a response in mid-sentence or mid-thought

I'm curious if this part actually does anything to improve the output, my intuition says it doesn't help much. I don't have it running locally to test yet, though.

These prompts become “key pillars” of the LLM’s “identity”. If you are curious you should at look about how much personality MS Sydney acquired through it’s own startup prompt.

Lol, was that blank an accurate answer about cybernetics?

That's the prompt, it's where the inserts the stuff it generates. The AI generated content has not been posted.

Yes I believe the parent was making a joke :)

I think accidentally the bot is kind of correct, you are asking to "complete the following questions" and that question seems complete, maybe the bot could have asked for clarification, I've never had that situation yet, would be something like .. "That question seems very well formulated, do you want me to answer it ? or to complete the question? we could complete the question by being more specific about location, for example what happened to the filed of cybernetics in EEUU in the 1970s ?"

Why would there be medieval jousts in the 17th and 18th centuries?

What kind of effort does it take to have any idea what you guys are talking about? Is there a For Dummies book?

It’s essentially a question of did you take a look the code? There is comments in the codebase directly where the config values in question are being used. Eg.

> # repetition penalty from CTRL paper (https://arxiv.org/abs/1909.05858)

Now I’m assuming a base rate of knowledge for this to help, but in general I find diving into the code paths of open source models, usually is a good use of time.

there's no for dummies, this stuff is too new. there's a bunch of white papers to read; _attention is all you need_ is a good place to start, kinda.

if you troll HN for AI related stuff and just read read read, progressively googling the terms you don't understand, eventually the haze will lift and things will start to make some sense.

source: in the middle of doing that right now

> if you troll HN for AI related stuff and just read read read

Do you mean "troll" or "trawl"?

(I'm thinking of an old saying that the best way to get an answer isn't to ask a question, but to say the thing is impossible. Can't remember it well enough to google for the exact form, however).

IIRC, both forms work.

Trolling is done by hand with a fishing pole, sometimes on a dock.

Trawling is from the back of a boat, generally with a net.

Interesting. Until now I only knew the verb "troll" as in "being mean on the internet".

If I'm not hallucinating like GPT, I think the etymology of troll as in internet actually stems from trolling as in fishing.

I have a hazy recollection of folks in the early days of 4chan describing 'trolling for idiots' - as in, feigning ignorance to get a rise out of folks. Like, you post a dumb, obviously stupid but infuriating comment and watch people get all worked up about it. You know, for the lulz.

I feel like since then it's changed to just actually being a dumb asshole and enjoying when people are mad at you. Or maybe I've just gotten smarter. In any event at the time it certainly didn't seem like the kind of thing racist lunatics would go out of their way to do. Perhaps more the fool I, as I was a teenager at the time and as such that opinion cannot really be trusted.

I think the best way is to get a basic understanding of torch and then just reading what the code does. The model internals are hard (though for the transformer there are good step by steps online) but for this case it is basically playing with the output of the actual neural net inside the model. The net gives you a probability distribution to be next over all tokens (think words). Now it has proven to be useful not to pick the token with the highest probability but to employ different strategies. Top-k in this case means probabilistically sampling from the top k tokens instead of piking the best.

"Transformers for Natural Language Processing", Denis Rothman (Packt)

To understand how pytorch works (bottom-up with practice video lectures): Neural Networks: Zero to Hero Andrej Karpathy https://youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9Gv...

Having it be on par with DaVinci is a big claim, will definitely be putting that to the test

What does no top_p mean exactly? Set top_p to 0.0? or to 1.0?

1.0

BTW - Now that discussions about sampling techniques within NLP are going mainstream, I have to shamelessly self-plug my own research work in this direction, specifically in the context of Constrained Text Generation. Simply filtering tokens according to constraints before generating text is wonderful for fixing stupid failure cases in LLMs like ChatGPT - such as asking it to not write with a particular letter, asking it to respect syllable counts, or asking it to rhyme properly (look carefully, ChatGPT struggles with exact rhymes in poetry!)

https://paperswithcode.com/paper/most-language-models-can-be...

My work got cited by Gwern in this article! https://gwern.net/gpt-3

I love this. I first tried it with Grover in 2019 and ran into similar token issues: https://twitter.com/jonathanfly/status/1143773270451396608?l...

Looks like you don't have a 'most common/simple words' vocab restriction? That's a pretty fun one too. https://twitter.com/jonathanfly/status/1144345934857474048

I feel bad for not being active on Twitter, specifically because I missed out on stuff like this (which predates my paper). I should have cited this thread since it's evidence that someone else had utilized this technique in my paper!

I think this vocabulary filtering technique will pick up steam again now that good quality open source LLMs are proliferating. I've always wanted to try these techniques with ChatGPT, and the inability to do so has caused me to somewhat publicly complain about their lack of access to ChatGPT's vocabulary probability distribution (they don't give more than the top 10 most likely continuations).

You might be interested in this thread from 2019 here on HN as well, I linked to a line of work on constrained sequence generation from Sony CSL that is pretty nifty https://news.ycombinator.com/item?id=21334639

As an aside - do you have any academic references for masking / constraining based next step transitions? It's a trick I've been messing with for years, but I probably picked this up from some older poetry generation work. Haven't been able to find any direct reference for simple 1 step masking, let alone more involved things like constraining based on the unrolled, modified markov sequence or things bigger than 1 step.

I think that this is what you will be looking for

https://huggingface.co/blog/constrained-beam-search

This technique has been quietly available within HF for awhile, and has a lot of real world use-cases.

For any models where you get access to the code (counting HF in this), definitely a lot of tooling available - and this tool looks really nice hadn't seen it before! Hope similar levels of control can become available for the bigger, less accessible systems, even though people don't want to expose token log-likelihoods there should be ways.

I've also been reading your paper on these topics, based on previous comments on HN. So thanks for that! Always interested in work about constrained generation.

I'd really like to see some version of the techniques from this palindrome generation https://www.ijcai.org/Proceedings/15/Papers/353.pdf revisited in modern contexts. Or some of the other Sony CSL work on this topic (e.g. https://www.francoispachet.fr/wp-content/uploads/2021/01/pap...) which I will admit I (still) do not understand yet.

Thanks even the 13B model that runs way faster with top_p disabled works better!

What does "1/0.85" mean here? The reciprocal of 0.85 is ~1.176, so is that what you are recommending?

Give it few shot examples and the quality should improve a lot. If you want a bot able to do technical Q&A give it two questions as well as the answers in its prompt. The model will then infer it's role as an AI that answer questions truthfully, and will try it's best to continue along those lines

I gave it the prompt thats the example from here: https://huggingface.co/blog/few-shot-learning-gpt-neo-and-in...

Tweet: "I hate it when my phone battery dies." Sentiment: Negative ### Tweet: "My day has been " Sentiment: Positive ### Tweet: "This is the link to the article" Sentiment: Neutral ### Tweet: "This new music video was incredibile" Sentiment:

and it's output was:

Tweet: "I hate it when my phone battery dies." Sentiment: Negative ### Tweet: "My day has been 90% balls." Seentiment: Negative ### Twitter: "i don't have a favourite type of music i just hate country" Seentiment: Negative ### Twitter: "c'mon first sunshine of the year...why the rain already?!" Seentiment: Negative ### Twitter: "Dear twitter, are you getting tired of me yet? Seentiment: Negative ### Twitter: "I don't care about the race, I just care about the president being a psych

Doesn't feel like that did the trick, but at least you got me to setup the shell command in a way where I pipe long-form prompts in via a separate file.

I think the model only has pre-training on large text but not multi-task fine-tuning and RLHF. So it can be improved a lot.

Hopefully a RLHF fine tuned model will be released soon because that will make the technology much more useful and less disappointing to people trying to run an alternative of ChatGPT on their local machine.

It invented "Seentiment"?

The next version release now needs to be called "Drunk Librarian". I want it enshrined in law :-)

For people complaining that ChatGPT makes up stuff, I use to say it's like asking a librarian at gunpoint. Of course it's very polite, and it knows a lot, but it also doesn't have time to look anything up or do any real research.

Reply on news.ycombinator.com