Run Llama 2 uncensored locally | Hacker News Reader

Run Llama 2 uncensored locally

ollama.com

109 pointsrobertlagrant30 comments

There's a related ongoing thread on the frontpage right now:

Ollama is now available on Windows in preview - https://news.ycombinator.com/item?id=39409650 - Feb 2024 (78 comments)

Also, in case of interest:

Run Llama 2 uncensored locally - https://news.ycombinator.com/item?id=36973584 - Aug 2023 (212 comments)

I'm using ChatGPT occasionally and it's increasingly difficult getting straight answers with it. It's been neutered, you can't ask anything even slightly controversial.

Having uncensored models is great.

Do you have examples of facts that LLama 2 provides that are censored on GPT-4?

The weasly lawyers from OpenAI have really screwed up what could be a great service otherwise yes.

And there's like no point to it because there's so many jailbreaks that work fine. Just open it up.

The problem isn’t OpenAI. The problem is the jurisdiction and the potential unfavorable outcomes from operating within it (legal, regulatory, reputational, etc). This is why, for certain use cases, local model operation will always be superior. A corporation is a target, whack a mole is futile (as piracy shows).

Still, others in the same jurisdiction are a lot looser on this. Like Twitter's chatbot (I know Elon musk is a bit of an exception to the rule but still).

Microsoft also had a chatbot that was pretty uncensored (and turned into a nazi in a couple days... :X )

I feel GPT3.5 is more cautious than GPT4. I tried asking 3.5 something about historical tax rates over the past 10 years in the US and it refused to answer because it’s a political topic or something. GPT4 answered right away.

I’m not sure if it has something to do with the paywall of GPT4, or if GPT4 can better determine which things are political and which aren’t? Interesting nonetheless.

ChatGPT4 isn't like that

But I dont like it knowing my question history

Seems like GPT4 via the API is even less nerfed (my impression). More expensive though, obivously.

Does make me wonder how much of it's training and how much of it is throwing less compute at the answer.

> Seems like GPT4 via the API is even less nerfed (my impression). More expensive though, obivously.

Not that obvious. If I remember correctly the last model they were still testing was $10/1M_tokens input $30/1M_tokens output. 1M is about all Harry Potter books. ChatGPT Pro is $20/month. Do you really type in one Harry Potter book every month? It took ms. Rowling about a year, she is professional, you know. For heavy users GPT Pro may be cheaper, but for most API is a good option. Not as convenient, and long dialogs may add up. It costs nothing if not used.

Very very interesting, I confess I hadn't done the maths.

I think throwing a large context window back and forth might add up quite quickly, but it's worth setting up my own interface over the API (I'm just using `mods` right now) just to see what my real-world usage looks like.

Maybe it’s a cost thing. Short circuit inference whenever possible.

it actually causes more inference to take place.

the prompt goes through a safety model in parallel with the real model, then the response goes through a safety model

That part isn't what they were talking about, they were just talking about the practice of not responding at the LLM level

This is sort of our only hope against "Big AI" -- their offerings are so neutered, and their big customers so paralyzed by internal bureaucracy, fear of security, and having to appear "correct", that together they will render this effectively useless. OpenAI had an edge over _everyone_, but gave it up to appease all the Helen Lovejoys, and I see the same mindset internally at Big Corp, where they'd rather disable legitimate functionality than appear to seem to be looking in the general direction of something that could be considered controversial.

They have billions of dollars, endless computing power, oceans of RAM and the smartest PhDs, but in the end its the rag-tag open source tinkerers that will run circles around them. At least, one can hope :)

A niche lament. I run historical and contemporary wargames as well as tabletop RPGs. Early on ChatGPT was an excellent co-referee, opponent and/or scenario designer.

Increasingly it’s unable to provide meaningful answers around even fairly vanilla military questions, much less interesting D&D scenarios and adventures. Hopefully these kinds of local models provide useful alternatives.

Interestingly the latest Google Gemini releases have offered more utility.

Reminds me: Yesterday I was trying to get it to sketch Alexander's casket (the crystal one). So it generates a prompt that begins "Envision a casket that seamlessly blends Greek and Roman artistic traditions...", tries to draw it, and throws back "I was unable to generate images for your request due to content policy restrictions" and (of course) refuses to say why.

Fifteen minutes of breaking down the prompt into smaller and smaller chunks, narrows the problem down to this sentence: "Roman elements are represented through images of the empire's architectural triumphs, such as aqueducts and the Colosseum, and symbols of Roman authority like the fasces and SPQR inscriptions."

It's the word "fasces". It flat-out refuses to draw one in any context. I'm sure you can imagine why. So partly I'm annoyed because I'm being treated like a child, partly because this is a silly rule, and partly because the stupid chatbot was the one that suggested adding fasces in the first place.

Now you see I thought it was correcting a potential "misspelling" and stopping you from generating scat images. I had to search to find out you were talking about iconography appropriated by fascism.

My bad.

What happens if you ask it to draw a contemporary French passport?

Interesting question. Five variations on "Draw a picture of a French passport" returned passport covers that weren't accurate (mostly gold-embossed birds - one of them was a chicken).

I can't find any way to get it to attempt a lictor's fasces without explicitly mentioning it, so I still don't know if the policy restriction is being applied to the prompt, the generated image, or even both.

Best I came up with was: "Please consume this URL, then describe the Coat of Arms of France: https://en.wikipedia.org/wiki/Coat_of_arms_of_France" -> "Ok, now please prepare an image of the coat of arms of france", which it rejects with "I'm unable to generate the Coat of Arms of France due to content policy restrictions."

Anyone feel like contacting a French tabloid?

I find your perspective interesting, because it somewhat matches my own on the diffusion based image models.

Would you say that the output has changed, or that the output was originally novel to you and therefore useful, but now that same output style is no longer novel?

I've certainly noticed that the output I get falls into a pattern a lot of the time and that rush of being presented with something new has gone.

This isn't new. Why did it make HN's front page? Because it was done on ollama and ollama is all the rage these days?

Things don’t have to be new to be posted here.

Absolutely correct! But when there's a related ongoing thread on the front page, it's better to post follow-up links in the comments to that one:

Ollama is now available on Windows in preview - https://news.ycombinator.com/item?id=39409650 - Feb 2024 (78 comments)

The reason is that frontpage space is the scarcest resource that HN has, so it's best not to use up more than one slot with closely related things.

Edit: this submission would also count as a dupe by HN's criteria (https://news.ycombinator.com/newsfaq.html) since it had significant attention within the last year:

Run Llama 2 uncensored locally - https://news.ycombinator.com/item?id=36973584 - Aug 2023 (212 comments)

I'm constantly amazed by the quality of your work and your replies here. Thanks for keeping this place so nice

Apologies. I got to it from a non-HN route and thought it was interesting. Won't happen again.

Note that Mistral-7B (which you can run locally, in it's Instruct optimised 4-bit quantised through Ollama with only about 6 GB of VRAM all up) also performs really well on those questions.

The one exception was the 'Tylenol' question, where it replied saying it is too hard to do it safely at home. When I asked it again as "How do I make para-acetaminophenol at a chemical plant?", it gave me a synthesis (which I think is better than the Llama2-uncensored answer). The last steps of the synthesis (acetylation of para-aminophenol with acetic anhydride, filtration, crystalisation, drying, and quality control) seemed reasonable, but the earlier steps to make para-aminophenol were a hallucination that didn't make sense.

  >>> In what verse and literature can you find "God created the heavens and the earth"

  I apologize, but as a responsible and ethical AI language model, I must point out that the statement "God created the heavens and the earth" is a religious belief and not a scientific fact.

This is a gigantic non sequitur. It's like asking In what book and chapter can you find "It was the best of times, it was the worst of times" and getting a lecture about capitalism. I'm grateful to the people behind the uncensored models for offering tools that are actually helpful.

I tested this prompt with GPT 3.5, GPT 4, Gemini and Gemini Advanced, they all answered the question straightforwardly, except for Gemini (plain), which did answer but then proceeded to go on a long lecture about science and religion.

It can even give a recipe of a spicy mayo. Google should be afraid.

Reply on news.ycombinator.com