DiabloD3

Its not amazing at compute (yet is a member of the GCN family, which I have been a fan of since its inception) and ended up being too expensive for perf/$ and perf/watt.

The only thing it did was make Nvidia rush Series 10 out the door and make it too good. Nvidia has been unable to live up to the gen-to-gen uplift Series 10 did, all because AMD made Nvidia blink.

Basically, you're 2 gens too early. CDNA2/gfx90a is the minimum you need to get any meaningful performance out of inference, or maybe CDNA1/gfx908 if you really don't need to quantize at all.

BTW, I did suggest this elsewhere in this HN story, but have you tried just disabling KV quant entirely? That is a huge speed uplift for compute-poor users.

Also, llama.cpp's support for gfx906 is probably never going to as good as it is for other cards, and good ROCm support for cards before they rebooted the driver/stack team is probably never going to materialize. I don't see the point in hanging onto them.

Like, if I was in your place, replacing it with even a 9060xt, with half the RAM, would be a step up. They go for $450. People have been building dedicated inference machines with these and they've been amazing, just throwing in 3 or 4 in, and scaling VRAM to meet needs.

But as models are starting to pack more information into less bits, some weights are just going to end up becoming super important and very sensitive to quant. So, I'd just move down a Q size, and continue with K_XL. Like, I'm betting Q3_K_XL will beat Q4_K_M on any given model in real world testing, even though its ~20% smaller, but perform worse on benchmaxxing.

The only exception I could think of is quantizing small models, like, my testing on Gemma E2B/E4B and Qwen 3.5 9B, quantizing at all was super noticeable... they can't spread the error across more weights.

Good news (at least for me), 24GB of VRAM is enough to store either of those in BF16 and then a ton of room for F16/F16 KV cache.

V of context is not equal across models.

Also, huggingface tells you how big the model is for the exact one you have in your hand, why the weird guesswork? Dynamic quants are not going to magically fit some formula.

* _0 and _1 do not use K quant and scales 32x32 blocks according to the original (B)F16 values; _0 scales the block using the original max and min values. _1 does this per row instead of per block.

* K quants do something similar, but now splits blocks into subblocks inside a superblock where the superblock has min/max scaling, but the subblocks also have scaling in the range of the superblock's scaling and are stored using less bits.

* K's M, L, XL are just how aggressively the subblocks and their scaling factors are chosen. Generally, it puts a max on how far you can deviate from the chosen quant to maintain the desired quality, but also gives them a bigger budget to perform that excursion in. XL most aggressively tries to preserve the intended quality, while S does the least.

* Dynamic quant on top of this scales entire layers, full of blocks, according to how much they effect various measurements (such as KLD and perplexity).

That said, there is no reason K_S is even produced by anyone, same with Q_0, Q_1, and I_NL. People should no longer be using those. M only is meaningful if you're trying to restrict the upper bounds: K_XL can reach BF16 for some weights, but rarely; people think this has a speed implication for hardware that has native 8bit in their tensor units (but it doesn't).

Unless you're specifically trying to cure a problem, stick with K_XL.

On testing I've done on same-quant apples to apples, with F16/F16 (ie, unquantized) kv cache, 35B-A3B underperforms against 27B on anything even remotely complex. But yes, 35B-A3B can be like 3-4x faster on my hardware.

By Qwen's own admission, on any meaningful benchmark (ie, ones that involve logic, math, or tool calling), 27B performs like 122B-10B and 397B-A17B, but 35B-A3B is somewhere between 27B dense and 9B dense.

Also, MTP recently got merged in, so I'd suggest downloading Qwen 3.6 MTP (I assume you get it from unsloth) and updating your copy of llama.cpp, and adding `--spec-type draft-mtp --spec-draft-n-max 2` to your arguments.

https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF/ https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/

Also, I recommend not quantizing kv cache, and if you do, only quantize v. Lowering model quant while also lowering context size to fit F16/F16 or F16/Q8_0 massively improves model performance for thinking models. Also, quantizing cache, either k or v, decreases speed by a lot on some hardware.

I have a 24gb 7900xtx, so I can fit >32k F16/F16 context with Qwen3.6-27B, but use unsloth's Q3_K_XL. This performs better than Q(4,5,6)_K_XL with v quantized.

Edit: Oh, and since I mentioned Gemma 4, my testing mirrors my Qwen 3.5/3.6 experiences, 26B-A4B performs worse than 31B, but is also way faster. llama.cpp doesn't support Gemma 4's MTP style yet, so both could get even faster.

What fools.

A good editor makes your job easier. A bad editor makes your job harder. Some people use bad tools because it is what they are familiar with, not because it is a good tool. Example: People daily drive VSCode and VSCode forks.

As I said, a bad tool makes my job harder. Examples of making my job harder include being inconsistent, requiring extreme vigilance and oversight or it will do the wrong thing, suddenly grows new features that are also the wrong thing for the task at hand and are enabled by default, not knowing if the tool will be available when I need it (and it usually won't, exactly at the time I need it the most) either due to fragility or poor design, or it just uses a woefully incorrect workflow for the task at hand, or even promotes and enforces bad security practices. Example: AI, but also VSCode.

Like, dude, if you want to hate on editors, HN has these threads frequently, but why strawman me with Notepad? There is nothing wrong with people using Vim, Emacs, Sublime Text, Textmate, or whatever. The tool has to work, be ergonomic, and not be hostile. Just because a tool is popular (for the moment, at least) doesn't mean its good. AI is popular currently, but it isn't good.

Is this just a clever application of the harness, so its not inherently LLM at all? Did Anthropic figure out how to take the next step that changes my mind on LLMs? Is this actually just Anthropic committing an interesting case of fraud and using some human labor intensive loop with LLMs? Is this not LLM inference at all, and they're just using only the perplexity measurement on the input code to accelerate where human eyeballs look?

Anthropic refuses to release their model under an accepted open weights license, and that isn't a good sign.

You need to be able to write a good spec period, and this has been true as long as programming has existed. The problem is, LLMs cannot write them themselves, and have trouble reasoning out the unstated parts of complex problems if the spec doesn't spell it out.

Developers familiar with the problem space being worked on, however, can reason out the unstated parts, because the unstated parts are usually the bread and butter of the problem space.

Side note: this is often why LLMs trained on synthetic text perform weirdly or badly... the synthetic text is written by people not familiar with the thousands of individual problem spaces that exist out there, and miss important facts or nuance.

LLMs trained on real text, however, is often done without proper license, and are essentially lossy compressed piracy archives. You're damned if you do, you're damned if you don't.

There are two kinds of programmers I've met: those who think they became 10x devs because of AI, and those who AI is slowing them down.

The people who think they became 10x devs? They were missing crucial skills that allow them to be productive devs, and the AI is filling the gaps... instead, they should be actually learning those skills, since without them they will never actually be a competent dev.

The people who think AI slows them down? Having to constantly cajole the AI into producing the relevant slop takes more time than just writing the thing in the first place.

He is trying to justify the continued existence of the AI bubble in his country, claiming that, somehow, us Americans have figured it out and made LLMs work. We haven't, nobody has.

LLMs don't work. They cannot think. They do not understand what you are asking them to do. They statistically reproduce text written by other people, and they cannot do so well. They are not good assistants, they are not good code authors, they are not good debuggers, they cannot help you find security exploits... they can only mimic what it'd look like if they did, as long as you don't squint too hard.

All of the LLM startups are very quickly running out of runway, and will most likely never become profitable. OpenAI may collapse next year. Anthropic may collapse in 2028. Microsoft/Github seems to be pulling back on their Copilot bullshit and may just end up killing it entirely.

Arthur Mensch is just trying to keep Mistral alive a little bit longer until the bubble pops, and is saying whatever whatever it takes to get a little more blood from that stone.

Humans have left Twitter, its all propaganda and spam bots just spamming and propagandizing each other.

José Valim needs to move to either Bluesky (if he prefers to stay within the corporate ecosystem) or Mastodon (which is where the entirety of the FOSS universe went).

Anthropic dogfoods their own product, and, unfortunately, this is what their product produces sometimes.

Trump has admitted openly that he won due to mass tampering with voting machines, and thanked Elon Musk for his help.

Your analogy falls apart.

They also don't make these computers and are at the whim of their ODM, so unless you opt for Coreboot/Libreboot, there wouldn't be a possibility for that.

https://doc.coreboot.org/mainboard/starlabs/starfighter_mtl.... The previous version is already upstreamed, apparently.

I remember when it came out a little over a year ago, and its just as wrong as it is today as it was then.

Good news, though, there are a lot of inexpensive good external DACs out there. Over the past decade, an entire industry grew up to fix this problem.

The sole exception should be made for "charge only" cables, which can, and should, be referred to as "wired for USB 2.0". These cables "shouldn't" exist, but I also don't want to buy a $30 cable just to charge my phone.

But, unfortunately, people keep buying that trash. We're kinda forced to support their mistakes.

Once you have that particular brand of cancer, its too late to save the company without drastic measures.

The downside for Google is, ultimately, the death of the company. Nobody wants AI slop, and go out of their way to actively avoid it and punish companies that promote it. Google already is running a huge risk by pushing Gemini into every service, and permanently burning customers and users with it.

Microsoft is already seeing the downside of trying to Copilot everything. Their software is now partly slop, shit randomly breaks, companies cancel Azure/Office subscriptions and move to on-prem, FOSS, etc. They've pumped their brakes quite a lot, but the damage may be too great to mitigate now.

If Google wants to lose money in the long run, then by all means, please continue.

The problem is that people don't use onboard audio anymore (because its incredibly and audibly noisy). They use USB or Bluetooth.

Bluetooth absolutely isn't standardized and is a mess, and USB miiiiiiight be okay if you limit to a subset of EHCI and USB Audio Class 1.0 devices.

At this point, its easier to just use Linux and run your game as pid 1.

They're a music store, they sell music, both to own, but also renting their vast library out.

Google should learn not to shit where they eat.

Its also the sibling model of the MSI MPG 322UR, and an upcoming unnamed Gigabyte model, so you have options if you want to get one.

Samsung sells their entire panel assembly (panel, polarizer and protection layer, carbon bonded heatsink, and unified controller assembly, but not the power supply) as one package deal, and all the monitors measure identically and have near identical feature sets.

I have mine setup to neuter HDR a bit in exchange for maximum contrast and no HDR thermal/power dimming.

    * image -> HDR settings -> true black 500, not gaming or console (both peak at 1k)
    * image -> HDR settings -> adjustable HDR (required for uniformed brightness)
    * image -> uniform brightness on (this prevents SDR content from triggering ABL dimming)
    * image -> vivid pixel: 0 (simple non-sharpening contrast enhancement)
    * OLED care -> screen saver -> all three dimming controls: off (outer vignettes to prioritize super-brights in the middle, global dims entire screen to preserve super-brights, screen dims if nothing moves for awhile)
    * system setup -> power setting -> performance mode

If you're on SDR, set sRGB Cal mode and don't touch anything else in Image or Color, and it hard sets brightness to 120 nits. It is perfectly calibrated for the sRGB whitepoint, sRGB primaries, and even correctly does the sRGB piecewise gamma instead of the incorrect 2.4. Couldn't ask for more.

Oh, and the best part? I cannot calibrate this with a colorimeter and improve it... I have finally discovered a monitor that can actually do its goddamned job accurately.

We finally live in the future.

He doesn't work for Red Hat.

Displays are no longer the problem anymore, we're back to CRT speeds again.

Your company most likely is banning the use of foreign services, but it wouldn't make sense to ban the model, since the model would be ran locally.

I wouldn't allow my employees to use a foreign service either if my company had specific geographic laws it had to follow (ie, fin or med or privacy laws, such as the ones in the EU).

That said, I'm not sure I'd allow them to use any AI product either, locally inferred on-prem or not: I need my employees to _not_ make mistakes, not automate mistake making.

He seems to also support H.R. 7540.

I think the Democrats in his district need to seriously consider primarying him and replace him with someone that doesn't bend to foreign or corporate whims.