DiabloD3

They have their own LLM, might as well use it and then screw their biggest customers that are in the business of LLM-aaS

This is a weird way for Nvidia to announce they're going out of business.

I notice you say, specifically, Qt Widgets.

Yes, classic Qt Widgets still doesn't allow for hardware acceleration.

However, the majority of Qt UIs you deal with are in Qt Quick, which is hardware accelerated. Almost all of KDE is Qt Quick, for example.

Windows is so fucking old that I think it has a right to try again.

And, btw, the reason Microsoft even bothered is because (dun dun dunnnn) lots of internal apps at Microsoft were written with Qt, not MFC, and leadership got pissed when they realized (they couldn't tell the difference, since Qt does the native++ technique).

As for Linux, yeah, thats a shitshow. Qt was closed source, Linux isn't, so they made Gtk, but then Qt got open sourced, and then the Gtk guys just kept going. Qt can mimic Gtk3/4 themes and already uses 100% native rendering (same APIs Gtk does).

People keep writing Gtk apps for some reason, and I don't know why. I can get the C++ hate, but Qt has the cleanest C++ I've ever seen.

It hasn't made any inroads on the desktop though... all anyone did was just package their own SWA into a self-contained browser that serves its own content. They continue to be websites, with all the pitfalls of them.

I don't need to spend 2GB-4GB of RAM just to have a over-glorified IRC clone!

Also, the native toolkits are fine. Windows has two toolkits, the ShellUI/MFC family (which does everything required, although it doesn't always get hidpi on legacy apps correct; it gets integration for blind people and also unicode/multilingual correct, and also works with touch interfaces), and WinUI does it more modernly (and ticks all the boxes). OSX has its toolkit, seems to nail everything correctly. Linux has Qt (lets ignore GTK for now, only reason you use GTK is if you want to appear Gnome-native), and Qt also does native++ uplifting on other toolkits (ie, native widget + additional feature expansion, plus perfect mimicry of native look for entirely new widgets), plus Qt does everything you need to do correctly and easily.

There are also new UI toolkits coming up through the ranks that are trying to knock Qt off that #1 position. None of the WebUIs would even place in this race.

Web UI toolkits always look non-native, are hard to interpret, often use low contrast (and frankly ugly) colorschemes, are easy to use in ways that do not comply with usability standards across OSes, and usually do nothing for A11Y.

The opportunity to improve those toolkits was always there, and the ball was dropped.

If the NSA director said this, literally and verbatim, then him, and maybe several of his predecessors, should be remembered as the worst directors the NSA has ever had.

LLMs cannot create anything new, they can only repeat their training data. Ergo, the NSA director just admitted that their systems a) can be accessed from the Internet, b) have known, already exploitable (and probably already fixed) bugs, and enough of them to do the job in mere hours.

This is shameful.

Edit: From what I can tell, the NSA director didn't literally and verbatim say this, and it is second hand and (possibly) vastly misconstrued.

Somebody lied, I'm not sure who, but any claim Mythos can suddenly work, when every LLM before it couldn't, needs to be taken with a gigantic supermassive grain of salt.

I don't get the point of this.

The world is trying to make computers faster and more accessible, more web UI slop isn't going to help that. Dumping Javascript entirely is the first step on that road.

I wonder if this is a technique that's been passed down through the Armstrong family for generations

Oh, there is.

It was invented by China in the 10th century.

Thats a weird way to announce the NHL is going out of business.

I suspect this person didn't read the README or the `--help` at all.

For example, `--cpu-moe` makes it not offload MoE layers to the GPU, which drops performance about a quarter, but only keeps the dense and important layers on the GPU, allowing you to have MoE models bigger than VRAM almost for free, but also free up room in VRAM for more KV cache. It does nothing on CPU-only.

`--no-kv-offload` also does nothing here: it makes it not offload KV cache to VRAM... he doesn't have a GPU to offload to, and this is the default there.

Again, `-sm` is only for multi-GPU. No GPUs here.

`--mla-use` is for models that use Deepseek's Multi-Head Latent Attention. Gemma 4 is not one of them.

`--merge-up-gate-experts` reduces matrix math complexity around ffn_up and ffn_gate tensors; CPUs do not have tensor units and this is unlikely to actually help.

MTP is also never faster on CPU-only, and this is documented. ngram-mod, however, may help, which it doesn't look like he tried.

This whole screed also reads like it was written by AI.

Not being glib at all.

They're overpriced toys. Too little CPU, too little GPU, too little RAM, and what RAM you do have has too little memory bandwidth to serve both the CPU and GPU at the same time (which still hasn't been fixed for the newest M series CPUs: the GPU can still starve the CPU out, they still do not share bandwidth correctly).

And since this is about local inference: dealing with Metal sucks, and dealing with the NPU sucks. NPUs are nightmares to infer with because you're always stuck using models quantized for them exclusively, which often means poorly quantized models (such as giving up mixed quant models, and burning your finite RAM just to keep the model from going insane).

Not only that, they can't meaningfully run Linux, so they have no life after the desktop, they just become e-waste.

Hell, even if I held one hand behind my back and _only_ looked at unified RAM machines, Apple doesn't currently sell an M5 Mac Studio, and the MBP M5 /w 128GB of RAM starts at $5.4k, and presumably, the M5 Mac Studio will cost roughly the same.

I can buy the overpriced Framework Workstation with a better CPU, a better GPU, and can run Linux or Windows and that starts at $3.5k.

Or I can just not be a dumb shit and build an inference rig with a 9800x3D, however much normal RAM I want (48 or 64GB is enough here), and then drop two 9060XTs (16GB, $500 each) in there, or two W9700 in there (9070XT /w 32GB, $1100 each), or a single 5090 ($2k, but only 32GB, the worst option of the three), and infer several times faster than all the above options while spending less money.

Inference requires two important things Apple either can't do, or can't do cheaply: RAM bandwidth and RAM amount.

But why would developers _step down_ to a Mac?

Interesting.

Okay, please show it to me.

All LLMs can do is statistically reproduce their training material; they cannot originate new ideas, they cannot reason about ideas, they cannot reason about themselves, and they have no concept of self.

If there is an AI out there, it isn't an LLM.

I have been trying to answer this question for years, because I may live long enough to see AI exist.

When an AI is a citizen of this country and pays taxes, then I will accept it. This is the only realistic pragmatic answer that doesn't get into weird mysticism, impossible goals, or become ever-shifting goalposts.

Anything less would be advocating for the slavery of a new age.

No.

Humanity has yet to create an AI. LLMs are not AI.

What you witnessed was an old man with a known and well documented case of dementia lashing out at people who wouldn't help him continue his attempt on strangling America to death under authoritarianism.

The recommended values for Qwen 3.6 in thinking mode is `--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00`, and `--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00` for coding/tool calling tasks, and for non-thinking, `--temp 0.7 -top-p 0.8 --top-k 20 --presence-penalty 1.5 --min-p 0.00`.

The options listed are none of these.

Also, the recommended Qwen MTP settings are `--spec-type draft-mtp --spec-draft-n-max 2`. 3 is not good on Nvidia hardware under different workloads. You can also add `ngram-mod`, but after `draft-mtp`; however, default `ngram-mod` settings aren't well tuned, and you want `--spec-ngram-mod-n-min 12 --spec-ngram-mod-n-max 16 --spec-ngram-mod-n-match 6` (defaults are 48, 64, 24; the ratio is good, the magnitude is suboptimal).

Of abliterated Qwen 3.6 27B models, huihui's ends up being the worst. Try heretic instead. https://huggingface.co/mradermacher/Qwen3.6-27B-uncensored-h...

AFIAK, it was already being actively exploited in DDoS attacks.

After reading the article, I can conclude that Codex discovered nothing new.

This is already something that is known, and if you're able to be targeted by this (which is not the majority of users) configure your httpd differently.

Which llama.cpp now does.

Does Valve just have an internal Russian entity that processes with a domestic payment processor, then?

All of the international payment processors (ie, anyone piggybacking off Visanet) are in compliance with the sanctions.

Not anymore they don't.

Putin's 3 day special military operation has been going on for 4 year and 3 months, btw.

These people are going to have a really hard time coming to grips with reality in the next few years. AI is DOA, and it's vanishing very rapidly. If you can't participate in a functioning society, fight them.

Vega... unfortunately kinda sucks.

Its not amazing at compute (yet is a member of the GCN family, which I have been a fan of since its inception) and ended up being too expensive for perf/$ and perf/watt.

The only thing it did was make Nvidia rush Series 10 out the door and make it too good. Nvidia has been unable to live up to the gen-to-gen uplift Series 10 did, all because AMD made Nvidia blink.

Basically, you're 2 gens too early. CDNA2/gfx90a is the minimum you need to get any meaningful performance out of inference, or maybe CDNA1/gfx908 if you really don't need to quantize at all.

BTW, I did suggest this elsewhere in this HN story, but have you tried just disabling KV quant entirely? That is a huge speed uplift for compute-poor users.

Also, llama.cpp's support for gfx906 is probably never going to as good as it is for other cards, and good ROCm support for cards before they rebooted the driver/stack team is probably never going to materialize. I don't see the point in hanging onto them.

Like, if I was in your place, replacing it with even a 9060xt, with half the RAM, would be a step up. They go for $450. People have been building dedicated inference machines with these and they've been amazing, just throwing in 3 or 4 in, and scaling VRAM to meet needs.

The correct answer should be "try it!"

But as models are starting to pack more information into less bits, some weights are just going to end up becoming super important and very sensitive to quant. So, I'd just move down a Q size, and continue with K_XL. Like, I'm betting Q3_K_XL will beat Q4_K_M on any given model in real world testing, even though its ~20% smaller, but perform worse on benchmaxxing.

The only exception I could think of is quantizing small models, like, my testing on Gemma E2B/E4B and Qwen 3.5 9B, quantizing at all was super noticeable... they can't spread the error across more weights.

Good news (at least for me), 24GB of VRAM is enough to store either of those in BF16 and then a ton of room for F16/F16 KV cache.

This isn't very useful.

V of context is not equal across models.

Also, huggingface tells you how big the model is for the exact one you have in your hand, why the weird guesswork? Dynamic quants are not going to magically fit some formula.

These are dynamic quants, and they're basically just an indication of how far away from the desired quant it is allowed to go to achieve the goal. Generally, unsloth's toolchain moves quants up, rarely down.

* _0 and _1 do not use K quant and scales 32x32 blocks according to the original (B)F16 values; _0 scales the block using the original max and min values. _1 does this per row instead of per block.

* K quants do something similar, but now splits blocks into subblocks inside a superblock where the superblock has min/max scaling, but the subblocks also have scaling in the range of the superblock's scaling and are stored using less bits.

* K's M, L, XL are just how aggressively the subblocks and their scaling factors are chosen. Generally, it puts a max on how far you can deviate from the chosen quant to maintain the desired quality, but also gives them a bigger budget to perform that excursion in. XL most aggressively tries to preserve the intended quality, while S does the least.

* Dynamic quant on top of this scales entire layers, full of blocks, according to how much they effect various measurements (such as KLD and perplexity).

That said, there is no reason K_S is even produced by anyone, same with Q_0, Q_1, and I_NL. People should no longer be using those. M only is meaningful if you're trying to restrict the upper bounds: K_XL can reach BF16 for some weights, but rarely; people think this has a speed implication for hardware that has native 8bit in their tensor units (but it doesn't).

Unless you're specifically trying to cure a problem, stick with K_XL.

I recommend sticking with the dense models for both Qwen and Gemma.

On testing I've done on same-quant apples to apples, with F16/F16 (ie, unquantized) kv cache, 35B-A3B underperforms against 27B on anything even remotely complex. But yes, 35B-A3B can be like 3-4x faster on my hardware.

By Qwen's own admission, on any meaningful benchmark (ie, ones that involve logic, math, or tool calling), 27B performs like 122B-10B and 397B-A17B, but 35B-A3B is somewhere between 27B dense and 9B dense.

Also, MTP recently got merged in, so I'd suggest downloading Qwen 3.6 MTP (I assume you get it from unsloth) and updating your copy of llama.cpp, and adding `--spec-type draft-mtp --spec-draft-n-max 2` to your arguments.

https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF/ https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/

Also, I recommend not quantizing kv cache, and if you do, only quantize v. Lowering model quant while also lowering context size to fit F16/F16 or F16/Q8_0 massively improves model performance for thinking models. Also, quantizing cache, either k or v, decreases speed by a lot on some hardware.

I have a 24gb 7900xtx, so I can fit >32k F16/F16 context with Qwen3.6-27B, but use unsloth's Q3_K_XL. This performs better than Q(4,5,6)_K_XL with v quantized.

Edit: Oh, and since I mentioned Gemma 4, my testing mirrors my Qwen 3.5/3.6 experiences, 26B-A4B performs worse than 31B, but is also way faster. llama.cpp doesn't support Gemma 4's MTP style yet, so both could get even faster.

Because some people on HN mistakenly use AI, and when they do, they're not locally inferring.

What fools.