Hacker News Reader

Comment by pixelpoet | Hacker News Reader

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4

Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM

With a bit more speed and model improvements, local AI becomes a reasonable practical thing! The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately...

I got my Strix machines for ~2k eur each, best computers this 90s kid has ever owned, but those days are gone :(

View on HN

I had a Strix Halo laptop with 128GB which unfortunately died last week. I paid 2800 euro for it. If I buy the same machine today, the sticker price is 7899.

The device was not perfect by any means, but the ability to run fairly large models is some kind of magic.

>sticker price is 7899.

It's not even worth it at that point.

You can get a used enterprise grade SXM baseboard with 4-8 V100/A100 GPUs off eBay at a similar price. That will even get you actual HMB ram and NVlink. Along with 10x the AI performance, assuming you don't care about your electricity bill of course.

You can get a new M5 Max MacBook pro with 128 GB unified ram (targeted by Antirez for DwarfStar4) even after the Apple price increases, it's less than 7899 by at least $1000. And you probably won't pull more than 100 Watts.

You are comparing US pricing with EU pricing. EU pricing includes 21% VAT and currency conversion "rounding up".

The cheapest 128GB Macbook Pro here costs €7.949,00.

No doubt a better value than the HP, and will depreciate a lot less quickly, but just as expensive. Unfortunately, not being able to run Linux is a breaking point for me.

I have one of these. Got it a few weeks before the price increases. On the 14" version charging is limited to 96 watts, but the chip can pull north of that with adequate cooling, so the battery will literally drain while plugged in.

It isn't a problem for me, more amusing than anything else (I run in Low Power mode 90% of the time) but worth knowing for anyone thats thinking about pushing the hardware to its limit 24/7.

> And you probably won't pull more than 100 Watts

I've played with ds4 on such a machine, and the battery drains when using only a 96W power brick. But you can put it in low power mode and the fans won't even turn on while it delivers something like a third of the performance.

ignoring the fact one would need a bit of a different setup (chassis, PSU) to run it, I casually looked and there's nothing below $25-50k euros for such a board decked out, depending on the config. TBH even that doesn't sound bad, but I wouldn't even know where to start how to run it.

This is from the USA site:

https://www.ebay.com/itm/157742745616

It's always a gamble buying used electric but per the description, fully decked out server with 256gb vram.

Though at 2U it is going to sound like a 747 taking off from your office.

So that’s where all the used V100s are going. I’ve been thinking about making a water cooled version of those because I don’t have a rack to put these servers on.

That's surprisingly inexpensive for 256 GiB of VRAM, but how much power would it take to run it? Is it even feasible with residential circuits?

Yeah it isn't worth it, but comparing a server with a laptop is also not a relevant comparison.

I didn't get a Strix Halo laptop because it was the best bang per buck, I got it because it was an awesome machine that could do a little bit of everything, fit in a backpack and only needed 140W.

But noone should buy one at 7899, obviously. It was a tough sell for me at the old 2800 pricing.

Can you link because 8k is the price of two A100s so that would be a steal.

4x A100s with the baseboard for $9k (USD).

https://www.ebay.com/itm/336632412718

Buy one and let me use it when you're sleeping.

how do you even connect these? i assuming expensively

The connection does not look standard so probably need to find some kind of OEM interposer card, which would allow you to connect some slimSAS cables that in turn connect to pcie adapter cards.

You'd need at least threadripper cpu/motherboard to handle that many pcie slots/lanes.

Not to mention that a baseboard like that is not going to work with a standard ATX power supply, so you'd have to provide your own power solution.

If that doesn't sound fun to you, you're probably better off buying Claude tokens.

first off rude, secondly, the issue is the existence of a mcio to whatever mess that thing is. i assume it is sxm5 or something and you probably need a pci-e 6/5 switch in the middle

everything strix halo went 2-3x bananas, same ballpark figures as apple hardware now and lead times on all of those are in months. Ridiculous where we ended up at.

The Bosgame M5 128GB went up 52% in price ($1840 -> $2799), so not 2-3x. And they are in stock. Still, a very big price increase.

For that price surely you get the rtx pro with like 96GB and vastly superior compute?

Yes but it depends what you want. I didn't want to spend 8k (or now 11k) for a video card that I couldn't run without turning my home office into a boiler room and pay through the nose in electricity.

I wanted a laptop that could run some medium sized LLMs locally and experiment with. Strix Halo is great for that. But it was also 2800 euros for a 128GB premium laptop. Not 7899 like it is now.

Shame it broke out of warranty.

Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k.

Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.

> Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4

Only if you pay the Framework premium.

https://www.bosgamepc.com/products/bosgame-m5-ai-mini-deskto...

I don't have access to the USD price, but it's 2500€ (tax included), up from 1600€ in November when I ordered mine.

that does not have thunderbolt / usb 5 though

Also probably an unknown china firmware instead of a supported one?

(Also no nazi bar)

nazi bar?

https://community.frame.work/t/framework-supporting-far-righ...

lmao no.

I think people buying laptops for AI use are, sorry, just plain crazy. You overpay for the screen and keyboard and battery and whatever, plus you get much worse thermal performance because of basic physics (area vs volume). My Framework Desktop has a Noctua cooler which works really well.

[Tangent: all my life I've been downvoted into a smoking hole in the ground, particularly on reddit r/hardware, for questioning the wisdom of laptops for high performance computing, including gaming. Everyone insists they need the mobility, and then just leave it plugged in the whole time, absolutely refusing to admit it's about aesthetic preference.]

I’m mostly with you but there are some people who like to use one machine for both laptop and AI work, and it’s much cheaper than buying two separate devices.

I generally agree for everything except Macbook Pros which outperform most available desktop setups for AI tasks - but they are also now out of reach for most people after the price hikes (6.7k now for 128gb, i got mine for 4.7k just about a year ago).

Honestly I think this is just a bad time to be buying hardware - everything is marked up an insane amount that doesn't really make sense.

For me the smaller footprint, lower power consumption and portability (admittedly between desks only) are the three advantages of using a laptop over a desktop for these purposes.

The Strix Halo mini PCs use the exact same chip, and have a much smaller footprint than any laptop. Have you seen the size of these machines? I can and have easily popped my daily driver computer into my very small backpack to attend a demoparty for example.

With the laptop you probably won't get silent operation at the peak 100-140w, i.e. you've now massively overpaid for lower performance.

Can you get these from vendors like Asus and lenovo these days?

The ones I've seen on aliexpress are from unknown Chinese vendors.

I have a Framework Desktop as primary PC (great cooling, beautiful case with handle) and the Bosgame M5 dedicated for AI use.

I was also a bit wary about Bosgame but TBH they've been great and the machine is rock solid, if a little noisier than and not as pretty as the FD. You can just buy from them directly and be fine, best computer deal out there by a mile.

I've had an ASUS PN-50E minipc, and its bios is broken and ASUS won't fix it, it cannot wake from sleep on Windown 11 25H2. Support told me to reinstall windows... and never got past level 1 support.

I've also got a Beelink minipc, which also had some issues, and the unknown Chineese vendor did respond meaningfully to support email, and created a BIOS update in ~2 weeks after a driver update surfaced a firmware bug.

I'd recommend skipping ASUS. Lenovo may be fine.

out of edit time: so my main point: Beelink may be new to the business and relatively unknown, but my overall experience is very positive with them so far. Fr better than with some Big Brands.

Maybe there is a reason?

It's like you are advocating for a public transport instead of a personal car but when questioned how to get to a place which is not erviced by a public network your solution is to rent a bus.

The cheapest ones with 128GB were 1580€/$1840 as late as mid December.

What's the advantage of ds4 over llama.cpp, esp if down the line they upstream his forked kernels?

Currently, llama.cpp clusters don't support tensor parallelism, have a look at Donato Capitella's detailed report: https://m.youtube.com/watch?v=PkKXm_mKCCM He also provides rocm toolboxes for Strix Halo: https://strix-halo-toolboxes.com/#about

IIRC llama.cpp doesn't implement DSv4's compressed attention mechanism, and while it does use (credited) parts of llama.cpp, it's focused on this great model for now. Much of this is covered better in the repo's readme.

In repo Readme and antirez reddit comments there was also expressed willingness to upstream.

I think mainly that he can move much faster with specific improvements targeting Deepseek on Systems with unified memory (Mac or Strix). It's a lot easier to optimize if you don't need to worry about all the other architectures. So optimize he did and it's just a lot faster than llama cpp for deepseek v4 pro and flash. Also interesting features are more doable, like SSD streaming, which makes it possible to load MOE weights for a model larger than your VRAM, I don't see that landing in llama cpp anytime soon.

I was hoping to buy a competent local model machine later this year but given the prices I’m shelving that for now. Especially because the frontier models are very cheap relative to the cost of building my own setup. Especially because AI specialised hardware and processors are improving very fast, meaning hardware we buy now will become obsolete for this use case much faster than for traditional computer use cases.

In 1-3 years the hardware crunch will be over, local distilled models will provide Opus 4.8 like intelligence, and the hardware will exist to provide usable performance.

Are they fairly reasonable performance from your experience?

I'm about to move house and one of the things I want is better local AI. Ideally running a variety of smaller models at the same time, but occasionally one larger one. 128GB unified is pretty appealing, but for the price I can also get a handful of 5060 ti 16GB which give vastly better compute/inference speed.

Seen a few reviews of the strix halo boards from people complaining the tps is too low so definitely keen to hear others' experiences with em.

>The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately...

You realize "tech companies" isn't a monolith? Micron charging inflated prices doesn't magically benefit OpenAI. The "high prices keep out competitors" theory doesn't make much sense either. It's like saying Dennys benefits from higher egg prices because it makes cooking eggs at home more expensive.

You got it wrong. Use appliances instead of eggs. If getting an oven gets more expensive I rather keep going to Dennys.

It’s classic capex vs opex. I’d keep paying my openai subscription instead of dropping $3k to run a subpar model. If the thing costs $1k I would consider it.

The big AI companies already have all of the cheap hardware to run on. It's more like if Denny's bought up all of the eggs when prices were normal, held onto them, then kept buying to keep egg prices up.

>The big AI companies already have all of the cheap hardware to run on.

Have they? Aren't they doing a massive datacenter build-out right now? Moreover the massive profits for Micron and Nvidia must be coming from somewhere, and I doubt it's price-sensitive consumers.

Dennys can benefit from higher egg prices if they can lock in long term contracts with suppliers for lower egg prices when smaller companies selling directly to consumers can't.

I think that realistically, companies compete against each other as individuals and compete against smaller companies and individuals acting more like cartels/monopolies, and that's what OP is referring to in terms of hardware purchasing/contacts/pricing. This also extends outside of tech to investing, so it's likely not just tech responsible for this.

openai etc are going to have a higher utilisation of the hardware so can afford it more than small companies/people. Efficient resource use matters more when they're expensive.

if strix was 2k now i would insta buy 1

Check https://news.ycombinator.com/item?id=48705076

The top model is a little over 2k, though.

Reply on news.ycombinator.com