Hacker News Reader

Comment by vlovich123 | Hacker News Reader

I have compared instruction following of stock gpt-OSS against stock qwen and the 20B outperformed all others, intelligently following instructions and reasoning about tool calling correctly for tools it hasn’t been trained on. Additionally, it performs like a 3B model because it uses 32 experts. I don’t know where this claim that it sucks comes from but my evaluation of similarly competitive models showed it leading the pack by a lot.

View on HN

The performance is great, but the censorship is ridiculous for me. I tried it as a backend for my game Guessix[1], but it would refuse for ridiculous reasons like "Cannot answer questions about copyrighted works like Harry Potter."

1. https://guessix.com/

Use constrained generation

Mind explaining?

Curious as well

If you constrain the model to a JSON schema, most frivolous refusals go away.

And if you finetune on a few formatted examples the effect is even greater

Do you mean like structured outputs? Unfortunately here the model is guided to explicitly tell you when you violate the rules and why, it can confuse it's system rules with the game rules and say you're not allowed to ask a question about copyrighted material etc.

Try the uncensored/jailbroken variants like openai-gpt-oss-20b-abliterated-uncensored-neo-imatrix

I just tried to ask it how to make crystal meth and it generated a very detailed step by step guide

I have heard that uncensorted gpt-oss is not very good because of it being trained mainly on synthetic data. Is not not true?

Iirc abliteration (ablation?) can be done without "training" and is pretty quick. It finds the individual weights related to the concept you want to ablate, and modifies those weights to "deactivate" them. Precision brain surgery, to anthropomorphize.

The problem with synthetic data would be that the censored information would not be in the training data at all.

Very interesting! Do the benchmarks hold up well or does it reduce performance in other areas too?

I'm always so confused by those statements as well. Because just like you, I feel that the 20B version is really good at following instructions.

Some of the qwen models are too, but they seem to need a bit more handholding.

This is of course just anecdotal from my end. And I've been slacking on keeping up with evals while testing at home

Even if you're right about all of this it still doesn't refute my "fine-tuning doesn't work for 99% of customers including you" thesis.

Also, most mainstream AI benchmarks do not agree with you:

LLMarena (https://lmarena.ai/leaderboard) has GPT-OSS 120B as #53 and GPT-OSS 20B as #69 (nice), which is extremely far from leading.

DeepSeek V3.1 is ranked #9, and is a solid 60+ elo points above GPT-OSS.

I know you're going to link some of the "ya but chatbot arena sucks cus of theoretical attacks against it" paper and the llama4 debacle, but here's more evidence that GPT-OSS blows:

https://livebench.ai/#/?q=GPT-oss

GPT-oss global average: 54.60

Deepseek V3.1 thinking global average: 70.75

Qwen 3 32B global average: 63.71

So bring receipts next time because I did.

Qwen 32B is a dense model. The competitors for GPT-OSS 20B are max 8B dense models. You’re comparing it against models it’s not competing with and calling it crap. That’s like saying Ferrari is better than Toyota. Sure, but only if we’re comparing 0-60. If we add a budget of 20K, suddenly the Toyota starts looking more competitive.

I never claimed it was a frontier model. Just best in class for the performance it can achieve and the memory footprint it can fit in.

And btw OSS did super well on domain specific tests without fine tuning. A model I don’t need to fine tune beats one that does.

You don't mention that MoE models horrible damage logprobs (by definition) and basically need a whole new theory of LLM sampling written for them.

Dense models are better for a reason, and the idea that "everyone is doing MoE now and dense models are dead" is total bunk nonsense.

You can quantize dense models, and 4 bit quantized Qwen 32B is still better than full precision GPT-OSS. Luckily Unsloth even gives you tools to go down to 1.58bits!

Excuse my ignorance, but what is a “dense” model? Why is comparing GPT-OSS 20B against 8B dense models “fair” but not when comparing it against 32B models?

So gpt-oss-20b is a sparse MoE model

Which means it has ~3b parameters active per token.

Qwen3-32b has 32b params active per token

An MoE model will not activate most of the model for any given query, so there is zero compute happening across an increasing part of the model.

Dense model means 32B parameters => 32B get used in calculation for every token. Every calculation takes time, and assuming similar latent space size (which they all have). For example Qwen-32B

MoE model has for example 80B parameters, but only 3B get used in calculation for any given token. For example Qwen3-Next 80B A3B

Performance comparison:

Qwen-32B => 56 tok/sec, 32 GB of VRAM

Qwen3-Next 80B A3B => 167 tok/sec, 85 GB of VRAM

So despite being close to 3x "bigger", Qwen3-Next is more than 3 times faster with the same compute capacity. There's a but though. But because what gets activated from one token to the next is a different subset of the model, it is still critical to have all 80B parameters loaded into memory.

So MoE performs much better with less compute, at the cost of more memory. It also performs better on benchmarks, it is a better model.

Similar techniques have long been used in ML to great success, rather than trying to create one brilliant model, create many that each have pros and cons, and then train a second model to figure out the best model for the task in front of you. There's even a name for the practice "ensemble models". MoE only kind-of fits (because you can't easily swap out models)

There's other factors, the big one being attention. That's why non-attention models, like MAMBA, will wipe the floor in terms of performance per flop (a compute unit), with anything else. When it comes to intelligence however ...

Qwen3 Next and Qwen3-30b-a3b are pretty decent proxies for GPT-OSS 120B and 20B respectively (and in fact are both MoEs with 3B active rather than 8B active parameters), and they lap GPT-OSS pretty hard in this specific benchmark, getting to #17 and #33 respectively. That being said, it's hard to take benchmarks beyond a grain of salt because real world tasks that I try to use these models for always have a lot more variation than the benchmarks illustrate. I do view GPT-OSS as a pretty good alternative to the Qwen models in some cases but there are tradeoffs -- while I see better reasoning from GPT-OSS sometimes, the prompt adherence and overall flexibility of the Qwen models makes them a lot better IMO as general purpose local open weight models.

Those models got released later than GPT-OSS. It’s like saying the Android phone released 6 months after the iPhone is faster. Maybe, but it also had 6 extra months of development.

I literally talked to 5 customers last week that needed fine tuning, legitimately needed it. I get if you’re just doing basic RAG on text you generally don’t but that’s only part of the ecosystem

I didn’t say it didn’t need it, just that it needed it less. For example, previous models needed a lot of careful fine tuning to properly do tool calling. OSS does not need that.

Reply on news.ycombinator.com