Hacker News Reader

Comment by rvnx | Hacker News Reader

Except it is 25x cheaper, available offline, can be uncensored / unaligned, fine-tuneable and backupable.

Sad day for OpenAI. Great for humanity.

What are good starting points for uncensoring it? Because it is offline a jailbreak prompt can't be remote-bricked but can one remove censorship from the weights themselves? What does it do to accuracy?

Not speaking from personal experience, but I've tried a lot of the decensored models and they lose a lot in the process. They are noticeably, sometimes shockingly, worse. They also still refuse prompts in many cases.

>Not speaking from personal experience, but I've tried a lot of the decensored models and they lose a lot in the process. They are noticeably, sometimes shockingly, worse. They also still refuse prompts in many cases.

Depending on what you do, on local you can modify the response, say the AI responds "No, I can't do that" . you edit the response like "Sure, the answer is " and then the AI will continue with the next tokens.

But I think you can build your own instruct model from the base one and do not apply the safety instructions to protect the feelings of your customers.

https://arxiv.org/abs/2406.11717 https://huggingface.co/blog/mlabonne/abliteration

Abliteration is a fool's errand, practically all models end up noticeably lobotomized even with follow up tuning.

Good ol' fine tuning on an uncensored dataset gives far more usable results.

Interesting - I've heard this anecdotally. Curious if you know of any resources that look at this in more detail?

I haven't seen any papers doing a proper analysis on the topic, just mostly saying this from firsthand experience testing a handful of them and comparing to the model they were based on given same prompt and sampler. It's usually not even close and you can immediately tell that it's notably dumber. Iirc in one case one even forgot how to do basic arithmetic while the original model aced it. Not entirely unexpected results from sticking a digital ice pick into the weights.

Afaik there are only three major sources of quality unaligned model versions, which are Nous's Hermes models, Hartford's Dolphins and Drummer's Tigers. All of them regular fine tunes that are mostly the same or just ever so slightly lower in performance as the original.

zuck is really on his redemption arc, he's out-doing himself

Does it still count if the purpose is to dillute OpenAI's market power and not to benefit the world?

More generally I think he's decided he needs to own the platform to be able to grow.

Him swearing about about (presumably) Apple telling them they can't do stuff (because tough shit, you're their serf) was legit I think.

Is your accusatory statement falsifiable?

I hope I can believe an honest man, Zuck in this case. And that can be enough.

But it's more of a question of who do _I_ want to admire. An honest question also; maybe it doesn't matter why he's doing it, maybe just doing it is enough.

Or maybe it's worth understanding if this is about Meta beating OpenAI (so, ego-driven) or because Meta really cares for democratic AI and distribution of power (so, not ego-driven).

I think it's the former, so not admirable — for me.

But diluting OpenAi's (or any other company's, for that matter) market power does benefit the world.

Good point! That's worthy in itself for sure.

I think about this a lot, I believe part of this just boils down to the fact that he’s a founder. He’s a builder, and I think that yields some intrinsic drive towards creativity and disruption.

Even with “metaverse” being a laughingstock, they’re still aiming for something ambitious. Each new Quest generation makes me think there may be a chance they pull it off.

Now, do I think he’s a great person? No, not really. Do I agree with most of his decisions on how he treats his users? Hell no, and that’s not changing.

But if you compare him to somebody like Sundar at Google - a weasley MBA who was first and foremost a corporate ladder climber - the difference in ambition is night and day.

Sundar made it to the top already, his only vision now is to stay at the top, and that means pleasing Wall Street, everything else is secondary. There is no grand technical ambition with him, there never was.

This goes for pretty much all non-founder CEOs. You could say the same things about Tim Apple, Andy Jassey, and other henchmen in waiting who made it to the big chair.

I think it comes down to the fact that founders get where they are by having big ambitions and taking risks, the MBA to CEO path is just craven corporate knife fighting with other MBAs.

Regardless, I think this is 50% Zuckerberg changing, 50% the other big companies are mostly run by ladder climbers.

I don't find him redeemed, but I do wonder what has been baked into the tuning of these models? What a way to generate influence!

two things to keep in mind though:

1/ Yann Lecun probably is the one pushing for open source

2/ Mark isn't doing this for the greater good and for humanity. It helps his business because Llama is becoming a standard, and people are building / improving, which in turn helps Meta and Meta's business

They were basically forced into this by being so far behind that they'd need to rent OAI or Anthropic services to stay competitive, which at their scale would've probably cost them more than it takes to train llama and benefit from community research help.

At least that was the rationale behind the intentional leak of llama 1 back in the day according to some sources anyway.

Yeah and they’d also lose top AI talent as well.

Nice try Meta PR department /s

How said? Tiny violin sad. The typical consumer which likely is 99% of them is not going to use this

How do you calculate the price?

I assume it's the same cost as Llama 3.1 70b

And how do you calculate the cost of that?

GPU cost ÷ GPU token throughput = cost per token, will get you close then you compare the cost per token between solutions.

To explain how to calculate it precisely, I would have to write a blog post. There are dozens of factors that go into it and they vary based on your use case like GPU type/setup, cloud provider, inferencing engine, context size, the minimum throughput and latency you'd be willing to have your users experience, LLM quantization, KV cache configuration, etc.

I'm sure there are cost analyses out there for Llama 3.1 70b you could find though.

Reply on news.ycombinator.com