bitexploder

Right? To get comparable brand and quality DDR5, which isn’t particularly great at AI anything it is ~$2000. All you had to do was start hoarding 3090 GPU and RAM in 2023. It is unhinged.

I have a 3 year old gaming system. RTX 4080 w/128GB of DDR5. It runs Qwen 38 Flash around 44-40 t/s with 128K context. It is on a specialized build that caches MoE experts and uses an optimized 3bit quant that basically is within a few points of the full 8 bit quant. In general, in casual benchmarking with Alibaba's endpoint I could not tell much of a difference. Overall this model is very good on long horizon agentic work. The main pain point for it is that its input processing speed is slow. Regardless, it gets meaningful work done.

I paid $500 for the RAM in Nov 2023 :)

There is a canonical hacker news post about Dropbox being useless and a pointless idea at one point :)

China reinvests twice as much as EU. In a very focused way. It is very different due to their control.

Simpler view for me: this is one of the most capital intense technologies to exist. Europe does not have enough capital to compete. China is building its own chips. It’s own everything. Silicon up. How do you compete with that. Only Google and maybe Aamazon is doing it domestically.

No, it is actually very good. Qwen Flash 3.8 Next is fine. But you need ~128GB of RAM to get it going and not a lot of people have that or can serve it very quickly. I have been running it on an old gaming system around 25 t/s to do overnight work and it is very strong, even at 3 bit quant.

Qwen Flash Next 3.8 … even at 3 bit quant it is very solid.

They are just pushing for favorable legal environment before the Anthropic IPO.

https://en.wikipedia.org/wiki/Chinese_room I think about this once in a while. At some point if it does the thing almost perfectly is it still not doing the thing?

Tesla V100? Would have to be a 3-ish bit quant in 64GB ram

Flash next only needs 64GB for core model inference. If you really wanted it.

Need 64GB vram, 900+ GB/s speed, and a lot of system ram (128GB). Seems feasible. Hmm.

Maybe older GPUs work.

MLX?

20 years later… I don’t get joy from hacking things. But the 20 year wisdom is a lot of the time it doesn’t matter if it is safe. Just know when it does matter and worry about that :)

Things are different now on a Mac. Many small improvements make Ollama genuinely decent for many models now.

Less. Probably 2-3K if you build right. Qwen 3.8 27B on constrained tasks is Opus 4.6-ish to me, it just doesn’t know enough, but when task is laid out just gets it done.

Comes down to how much of the ambiguity we expect out of the model.

OMP. Opinionated but completely configurable. Probably the beat to have a lot of batteries and let you uninstall what you don’t want. Sadly Anthropic forbids its use on their subscriptions.

I was kinda kidding because parent called it "blazing" fast lol. I didn't think it was Rust. Blazing fast software is reserved as a descriptor for Rust programs :)

It is entirely reasonable to run a model like DS Flash 4.1 locally. Not easily. But feasible. If you drop down 100b - 200b models they are far more feasible.

DS Flash 4.1 is close to Opus 5. Beats Opus 4.8 in all of my little evals. You can abliterate local models. The genie is out of the bottle. There is no going back. This is why you see Anthropic and others talking about slowing down the frontier. Because they realize that you can’t even stop local users at this point. We basically have the equivalent of frontier models within reach on local hardware that cost less than $10k

It isn’t even a matter of time. You can just do it right now if you know how and have a bit of hardware.

e: this raises serious questions for me btw. The most obvious lever is controlling hardware. Controlling models and their capability is over. Controlling access to models is hard. Controlling access to GPU and fast RAM is more feasible. In 3-5 years every nation will understand how to distill, build, and abliterate safeties in models. This technology is fundamentally not difficult to work with.

You just have to Manhattan Project it. Fable will happily work on almost any piece in isolation. The hardest boundaries are domain specific terms you just can’t avoid. It can be tricky. I have found a way around it mostly.

Or just use DeepSeek. I have had Flash 4.1 with Ghidra MCP going since 4.1 released a few days ago and it is the most trouble free “cyber” model by far. It is also very strong. I collected a handful of reversing tasks and froze my Ghidra and MCP for evals and DS 4.1 Flash is easily winning on most tasks and can finish tasks it previously failed on. Anyone can reverse software and find vulnerabilities now. Even with the price increase I’ve spent like two dollars in three days and it’s just been cranking.

It is Rust now?

That makes sense. I think I have approached becoming stuck. I reached a kind of extreme disassociated state once where I felt like I didn't exist and part of my brain started to panic. I meditate a lot. I reminded the part of me that was becoming anxious that we are okay because we are somehow still observing ourselves so we must still exist. That was apparently adequate to calm it lol. It is truly weird to negotiate with some part of your brain and see it work in real time. I think it was a particularly sticky day for my DMN and or some particular loop it was hanging on to? It is always hard to say.

We have pretty strong indications of when and why things go wrong. Openness vs rigid thinking are an axis that are very strong correlative factors. A lot more nuance but there is a lot we do know about risk factors so it isn’t a complete question mark.

“Inadvertently”.

Someone was probably sad when they saw a compiler work for the first time after years of writing assembly. AI is intellectually different, progressing, and has no visible horizon, but in my life technology never did in computing. The point is that technology changes often. I think having a couple of decades in tech makes the shock easier to absorb in many ways. And also harder regarding something so familiar becoming so rapidly irrelevant.

But I think your message is correct. The agents don't do the hard work of making things useful and reliable for humans. There is more to do now, not less, somehow. The universe has changed, but most of the problems we are solving have not.

Hah, np, stego in general is really cool :)

That is a better idea. Ingesting your corpus with a lot of traces that have semantic patterns. Semantic steganography that suffixes well to real math and science (and any) topics. <thinking> heh.

Problem is how do you convince the model and training profess it matters. A one off canary is very unlikely to survive in the final model state.

I am surprised at how well DSv4 flash does in the real world vs many benchmarks. You look at Flash 3.8 and it supposedly beats opus 5 and deepseek is far below.. but they were measuring efficiency, whatever that is… Something doesn’t add up for me on the published benches

I don't usually defend Apple products, but my AirPods Pro 2nd gen have been rock solid and taken insane abuse. I use them working on cars, runs, hiking, rain, shine, sauna. I drop them. They land in puddles. They smack concrete every couple of months. Sample size 1, but they are probably one of my favorite accessories and things I lug around with me. I use them a lot and if not love /really/ like them and nothing else has come close to the frictionless experience of them.

Although that may be in part due to vendor lock in and not sharing some of their connection handling API / tech.

Thanks... I have been trying to figure out some things. Been doing my own evals. Flash 3.8 does burn a lot more tokens on high. Interesting how smart and not smart it is. For personal use almost impossible to justify the cost of 3.8 Flash cost.