Nvidia DGX Spark as a daily driver | Hacker News Reader

Nvidia DGX Spark as a daily driver

daniel.lawrence.lu

102 pointsplun980 comments

I strongly considered it, but the one thing that scares me away from wanting to do the spark is you basically have to use nvidia's linux (from what I've read) and it doesn't appear the nvidia is interested in upstreaming their kernel changes.

I'm avoiding where possible buying electronics where support is controlled by the manufacturer and not me.

A good instinct. There are a lot of things a $500 AMD GPU can do in linux that a $5000 DGX cannot.

I know nothing about this topic but your comment piqued my curiosity. What would a $500 AMD GPU do better than DGX?

AMD has Mesa drivers for graphics, which are better-optimized than Nvidia's proprietary Linux Vulkan drivers. It can be fixed in software, but Nvidia's only barely started to catch up.

The focus for Nvidia's GPU stack on Linux is getting CUDA working, which means that some traditional raster features get neglected.

NVIDIA also cares about Hollywood and CAD/CAM shops.

I learnt three times that AMD isn't necessarily better in Linux as advertised, now I don't care any longer.

I'm not aware of any commercial CAD/CAM software running Linux that Nvidia would have a money interest in supporting. Which current CAD/CAM is running on Linux with graphics? The closest I'm aware of it FEA packages allowing remote calculation on Linux machines while the GUI runs on Windows. Is that what you're referring to?

BricsCAD, ARES Commander and Platicity come to mind.

A lot of Hollywood studios use renderers that jump straight to CUDA as their backend. Cycles is a popular one, as well as Renderman and Octane.

Linux is part of the VFX reference platform, and you forgot Hyperion, for example.

Although one of the ones it cannot do is have/address 128GB of video memory, so it depends on what you want to achieve.

AMD Strix Halo was half the price at launch, same 128GB unified memory and nearly identical memory bandwidth. It is x86, so run whichever OS your heart desires.

You give up 200GB networking, but anyone 'daily driving' one of these things are probably not needing to cluster multiples of them at high speed..

I'm seeing it for 4k and up for similar specs as the spark?

Yes, due to the huge increase in memory price and also exploiting the demand for local LLM inference, the price for a mini-PC with Strix Halo has become more than double in comparison with the first months after launch (when you could pay something like $1800 for a system with 128 GB).

For those who are interested mainly in GPU performance, the NVIDIA system may be preferable. Otherwise, the Ryzen CPU is significantly stronger in multi-threaded performance, much stronger when running applications that are aware of AVX-512, so Strix Halo is preferable when the CPU performance matters.

Also for things like video decoding or transcoding, the DGX Spark GPU is likely to be weak.

>You give up 200GB networking, but anyone 'daily driving' one of these things are probably not needing to cluster multiples of them at high speed..

I mean that's not as true as you'd think: in fact its the whole benefit IMO of the spark-alikes, and if you're not running TP=2 or above you're sort of wasting it's potential

Fair. Honest caveat - I keep seeing "a good instinct" everywhere now. Is this humans acquiring new phrases from Claude? Is there a name for this phenomenon yet?

If not I propose HLRF.

Nice

DGX does parallel inference with llamacpp/vLLM much better than AMD GPU at 128GB VRAM.

If I understand your comment correctly, I think he addresses that in the first few paragraphs:

> The DGX Spark runs “DGX OS” but it is in fact just plain old Ubuntu 24.04 with some additions. If you want, you can just install another Linux distribution easily (Fedora works well), although there may be a couple of weird bugs with the Realtek Ethernet driver so the NVIDIA version of the Linux kernel has a couple of patches. Unlike some other ARM devices, the DGX Spark is all ACPI rather than device tree based, so regular Linux builds for arm64 work just fine.

Well that is better than what I gleened.

I thought I'd read that the GPU needed extra kernel patches to properly work. If it's just the Ethernet driver that seems a lot more appealing.

I thought it was device tree as well, so great that it's actual ACPI.

Many other distros almost work out of the box (as in they boot and run without any modifications). The custom kernel patches you mentioned address mainly non-critical bugs such as a bug where the Realtek r8127 stops working after a reboot (but it works if you turn it off and on again) [1] [2]. I'd consider it in a way better state than trying to run other Linux distros on certain device tree-based devices like, say, Qualcomm Snapdragon machines. The regular NVIDIA drivers with the open source kernel modules work just fine. Talos Linux supports DGX Spark since version 1.12 [3]. I also know of people using Fedora and nixOS successfully.

[1] https://github.com/NVIDIA/NV-Kernels/compare/ea55925ab430f1e...

[2] https://forums.developer.nvidia.com/t/realtek-r8127-ethernet...

[3] https://github.com/siderolabs/talos/issues/12170

Why are those patches not in mainline? They appear to be generic enough not to give out any Nvidia secret sauce.

iirc the bug that the R8127 patch was addressing was fixed in the R8169 driver upstream in kernel 6.18 and later. Not super sure what other patches there are and what the status of those are...

I have a DGX and a Ryzen AI Max 395 - while I love both of them, there are a few critical things that leave the DGX in use, while the Ryzen "just" is my primary homelab server. The biggest thing is prefil numbers, and the performance impact of higher context sizes. Qwen 27b is a great model, nemotron is decent, gemma is workable. But all of them need reasonable context for reasonable outputs.

Unfortunitly, as others have noted, the DGX OS experience... sucks. My hope is that the RTX Spark (which looks to be the exact same stack, sans the high capacity network interface) will help this get a bit more attention, but nVidia's long long long war with the open source community is not helping. Focusing on mainlining kernel support would go a long way to getting the community to be supportive.

Of course, a massive regression just hit Linux 7+/7.1 plus for ROCm hosts, so it's just rough everywhere.

I do appreciate how Nvidia tries to say close to vanilla with Linux and Android (nvidia shield). Instead of trying to build a shitty moat like Samsung with all their garbage software.

If nvidia ever releases Android smartphone, I'd probably stand in line to get one.

After leaving a few of great-on-paper SoCs as paperweights I've learnt that I just don't want to deal with anyone's custom platform as I'll eventually be left with an outdated system that's annoying and time-consuming to maintain.

I thought they did that once.

hmmm... about 2010-ish:

NVIDIA Tegra mobile phones include the Motorola Droid X2, LG Optimus 2X, Samsung Galaxy R, HTCOne X, and Xiaomi Mi 3.

https://www.androidauthority.com/nvidia-phones-1234416/

It is funny how they make some really decent hardware but the uptake can be fairly limited.

Tegra range is a good example. Yes there is the Nintendo Switch but beyond that it become very limited. Surface RT, Ouya, Nvidia shield and then a few small bits here and there.

Somehow related: "The end of my AArch64 [Ampere Altra Q80-30] desktop experiment", https://news.ycombinator.com/item?id=48728599 / https://marcin.juszkiewicz.com.pl/2026/06/26/the-end-of-the-...

The Ampere cpu in that post has a "lack of single core CPU speed" despite ripping through compilation tasks with its 80 cores. But the DGX Spark's CPU has fairly decent single core speed and is more like Apple Silicon in this respect.

This is an interesting review. I have a Chinese strix halo box that's isnt available in the west (favm faex1) I've been able to do some ok graphical gen, or some decent agentic tasks as a fallback for when some of the APIs are overloaded during business hours, but nothing amazing for sure, and also not both at the same time. But here's the thing...it cost me 1800usd two months ago....and it's runs x86. I am struggling to see why people pay +2x more for the Arm Nvidia version, despite the slightly higher bandwidth it still does basically the same AI tasks and alot fewer high end general computing tasks... I like my box but I wouldn't find it useful enough to pay more than I did for it or get more of them and cluster for instance. Can anyone explain the allure of the Nvidia box, other than brand name?

> isnt available in the west [...] Can anyone explain the allure of the Nvidia box

The first part might answer the second one. Otherwise, the lack of CUDA and the nvidia ecosystem of tooling could also explain why it doesn't seem so interesting for AI tasks.

But strix halo boxes themselves are available, just not that one. And despite my concerns about what's said about Cuda and ROCm I have never had a problem running any model, for image or text or voice, the community has done great work in making things work. So the point of the question stands.

It's also interesting that the most high end Chinese equipment, both prosumer things like these boxes but also the Huawei professional stack is just not available in the places it would be most appreciated. Not sure if thats china tit for tat, or western "we don't want your commie hardware anyways"

But for a lot of people it sucks cause nobody should be paying 4.2k for this product. The value isn't there.

> I have never had a problem running any model, for image or text or voice, the community has done great work in making things work

There is a whole world of other tooling and stuff that isn't just for hobbyists to run inference with ML models, but also how to do profiling, debugging and gathering data when you run distributed workloads, and so on. The nsight toolkit seems miles ahead of the competition on other platforms, as just one example.

Also, you can use handheld PC 395+ 128GB with battery for (semi-){0,1}offline use cases, works great for C2 of cognitive SDR radio (among plenty of other use cases).

Yep, stuff is more likely to "just work" on NVidia. Example: pytorch

In most benchmarks, the Spark is also faster at the prompt processing / prefill phase.

It's simple, the 395+ Max Strix Halo you bought for $1800 is now a ~$4000 build (at least the AMD AI dev unit). If only we had time travel right? Either way, the Nvidia unit comes with Connect-X 7. That may or may not matter to you, but the hardware for that isn't cheap. In general the Nvidia cards also have better support for models. I know AMD is trying to catch up but anything except their datacenter cards do not seem to be getting a lot of attention.

AMD is barely trying. The 395+ Max Strix Halo was launched January 2025.

gfx1151 was not listed in the ROCm compatibility matrix for ROCm 7.2.4 [0]. This is the previous version of ROCm.

It's only finally received support in ROCm 7.14.0 [1]! It literally just started receiving support last week.

[0] https://rocm.docs.amd.com/en/docs-7.2.4/compatibility/compat...

[1] https://rocm.docs.amd.com/en/docs-7.14.0/compatibility/compa...

Is it to me or the iGPU of Ryzen 9950X is not supported by ROCm?

The igpu is a tiny RDNA2 GPU. Pretty certain the 9950x cpu will outperform it in compute tasks.

What if the software was written for ROCm?

Vulkan has worked for a long time on gfx1150/1151 and works well.

I purchased my Spark back in March before the RAM price increase. My use case is local self-sovereign AI in Healthcare (https://hcvc.net), so I haven't done any gaming. Its been a great driver for Hermes Agent, serving local MoE and dense models up to 122B. It now handles about 90% of my inference load (more complex devops/coding tasks are still routed to cloud). LoRa works well. All my models are open source. There is no NVIDIA dependency in my build. Its also a backend server for my PoC app (https://proforta.com). Zero issues so far. Extremely reliable and gives me straight path to production for VPS deployment.

Super-powerful (if rather pricy) Linux desktop that happens to play games while doing everything else... that man is living my dream :'D

Neat blog! I was intrigued by this bullet point mentioned in passing:

>my four hard drive USB 3.2 ZFS raidz2 array with four 24 TB drives

Can you speak more about this? Which USB array did you choose? How well does it work? I've been slowly planning a transition away from my power-hungry surplus enterprise gear in the 19" rack towards a smaller, quieter, lower power setup ... but storage is the real kicker right now. I have a 12x18TB array in raidz2 built into a 1U NAS case, and I just can't quite figure out a better way to package something like that. I would need three USB arrays if I want to reuse the existing drives, which I think I do given how expensive storage is today.

It's an Orico 9948C3 with four Seagate Barracuda 24TB drives. They were on sale last year [1].

Unfortunately, the enclosure doesn't work super well on Linux. There is a weird bug where the drives don't enumerate when I boot up my computer. This happens on both my x86_64 AMD machine running Linux, and on the DGX Spark. The solution is... simply power cycle the enclosure a couple of times by toggling the power button on it and then it works. Once all four drives show up in lsblk, I can `sudo zfs import ...` manually. This is really gross and annoying. Replacing the USB cable, flipping the USB-C cable 180 degrees, hot plugging it, etc, all didn't work, both on the DGX Spark and the other Linux machine. I've also read reports of it being unstable in UAS mode on Linux but I haven't found a big difference in stability between enabling UAS or falling back to usb-storage.

Once it starts up correctly though, the drives are fast. I store my huge amount of 100 megapixel photos on it.

The Seagate Barracudas are helium-filled HAMR/CMR drives and are apparently rebranded/binned Exos drives. They aren't rated for 24/7 use but then neither are the refurbished Exos drives.

[1] https://www.reddit.com/r/buildapcsales/comments/1p29pm8/hdd_...

Yeah... that's been my past experience with USB docks, at least any with more than one slot. I've never had great luck with them. "Can recover from power outage without being touched" is a key requirement for my NAS so I'll give that one a pass and stick with my "SATA drives directly attached to a SAS controller" strategy for now. Thanks for the reply.

As for the 24/7 use, yeah so be it. The I in RAID stands for Inexpensive. If they fail after 10 years at 24/7, so be it. I have drive level redundancy and frequent offsite backups of anything critical.

It's absolutely the stupidest solution, but SwitchBot Bot is a home-automation button pusher. They are quite cheap on ebay (especially compared to the price of hard drives) but still a face-meltingly stupid problem to have. https://us.switch-bot.com/products/switchbot-bot

I have quite a mix of 2x and 4x drive enclosures, and generally I don't notice anything particularly odd about them? Some enclosures have degraded smart or power management capabilities though. I really wish I'd noticed at the time & sent those ones back! What a waste! My favorite are some dual slot HighPoint toaster-style ones that have independent USB connections per drive, which rocks, but these are so rare & I almost never take advantage of it (replug on drive into a different mini-pc).

Please don’t buy a DGX Spark unless all three of these are true:

  - You value simplicity more than performance or price-to-performance.
  - You accept that the hardware will depreciate rapidly.
  - You’re prepared to buy two or four of them.

OR:

  - You want to run frontier models right now as cheaply as possible
  - You want to run high-parameter models on a 15a breaker/line

Otherwise, get a normal, high-bandwidth GPU.

A single Spark gives you roughly 115 GB of usable memory compared with the 24–32 GB found on many lower-cost GPUs. It's certainly a big increase, but in practice it does not unlock dramatically better models.

  - One Spark: More memory, but mostly enough for poor-quality, extremely low-bit quants of larger models.  
  - Two Sparks: Enough for mid-tier parameter models at reasonable quants, such as DeepSeek V4 Flash and HY3.
  - Four Sparks: Enough for GLM 5.2 at a reasonable quant.  You'll need a $1000+ switch too.

The problem is that Sparks are slow compared with almost everything else in their price range. Many factors affect inference speed, but memory bandwidth is one of the biggest. A $4,000-plus DGX Spark provides only 273 GB/s.

  | GPU                 | Memory bandwidth |           VRAM | Approx. price  |
  | ------------------- | ---------------: | -------------: | ------------:  |
  | DGX Spark           |         273 GB/s | ~115 GB usable |        $4,000+ |
  | RTX 5060            |         448 GB/s |          16 GB |          $600  |
  | Radeon AI Pro R9700 |         640 GB/s |          32 GB |        $1,200  |
  | RTX 4000 Pro        |         672 GB/s |          24 GB |        $2,300  |
  | RTX 4500 Pro        |         896 GB/s |          32 GB |        $3,500  |
  | RTX 3090            |         936 GB/s |          24 GB |        $1,200  |
  | RTX 5000 Pro        |       1,344 GB/s |          48 GB |        $6,000  |
  | RTX 5090            |       1,792 GB/s |          32 GB |        $4,000  |
  | RTX 6000 Pro        |       1,792 GB/s |          96 GB |       $12,000  |

Yes, the Spark has substantially more memory. But going from roughly 24 GB to 115 GB does not necessarily unlock substantially better model quality. In many cases, it only lets you load heavily compressed 2-bit versions of larger models, such as DeepSeek V4 Flash, with serious quality degradation.

24–32 GB is currently a sweet spot. Models such as Qwen 3.6 27B and 35B-A3B:

  - Perform far above what their parameter counts suggest.
  - Fit comfortably within 24–32 GB of VRAM at reasonable quantization levels.

A 4-bit quant of Qwen 3.6 27b (18 GB) will out-perform a 2-bit quant of DeepSeek v4 Flash (90gb).

Instead of the Spark, if I had a roughly $4,000 budget...

Assuming I already had a reasonably modern desktop:

  - One RTX 5090, RTX 5000 Pro, or RTX 4500 Pro.
  - Two RTX 3090s, RTX 4000 Pros, or R9700s, provided the motherboard can bifurcate two physical x16 slots into x8/x8.

If I were building a system from scratch:

  - A DDR4- or PCIe 4.0-era consumer CPU and motherboard that supports x8/x8 bifurcation.
  - Two RTX 3090s, RTX 4000 Pros, or R9700s.

If I were already planning to buy a new Mac:

  - A MacBook Pro M5 with 64 GB or 128 GB of unified memory.

For context, these are the systems I currently run:

  - EPYC Turin with four RTX 6000 Pro Max-Qs.
  - EPYC Milan with four RTX 3090s.
  - AM4 with two RTX 3090s.
  - AM4 with two RTX 3090s.
  - Intel Raptor Lake with two RTX 5060 Ti.
  - MacBook Pro M3 128GB Unified

I'm actually confused by the post - indeed why choosing the 27B dense and the 35B MoE models? Given the structure of the post I honestly have the impression that the Spark was purchased out of curiosity more than for a specific purpose (LLMs).

Ah yeah, your reaction makes sense. In the circles I run in, there is a lot of hype around Sparks for inference, so my gut reaction is to respond with this type of warning.

I did not intend to imply that the post author was advocating that they're great for inference, as they're obviously not.

The 27b dense is _remarkably_ better at coding tasks.

Especially at larger quanitizations (Q4 is pretty crap).

Actually, its ~119GB usable (I have one). You can shutdown a lot of unneeded services if you only use it remote which will trim a lot more fat. You can enable the RDP service if you don't want to sit in front of it but get a desktop interface.

The "shitty" network is 10Gb/s and wifi7! You get a twin QSFP28DDlol+++ (I jest) that each run at 200Gb/s - not for the casual home user but handy at work, although I "only" have 40Gb/s on my switches sigh. With and no switch two you can do a three node cluster with some careful networking. If you want to do more then a switch is needed and it will need to be pretty funky! That said you could wire them up in a circle and use VLANs and MSTP and accept less than 200Gb/s per link. You'll probably need Openvswitch and a lie down afterwards.

I'm not a fan of the Gnome desktop but it works well enough and I think the Nvidia customised Ubuntu is well thought out. You get all the complicated NVidia extras pre-installed, along with docker (full fat, not the Ubuntu one) for a fairly quick start. It includes Ubuntu Pro which is free for five systems anyway but its nice to see it pre-installed.

You can run quite decent models on this thing see: https://spark-arena.com/ Also see "DS4".

We blew abut £4000 on one and it will pay for itself in a few months. I tried pricing up an Apple thingie and the Store wouldn't offer me more than 96Gb of RAM and a delivery date in Q3 at the earliest. Our Spark rocked up next day. They seem to come in 1TB or 4TB SSD variants. 1TB is enough for me and saves a lot of cash - keep an eye on your model downloads and ruthlessly delete old experiments. docker system prune.

We went for the Asus variant that has active cooling and I stuck it in the ceiling cable tray over our computer room racks. It sits on 1½" stainless steel mesh with lots of clearance in an actively cooled environment.

That ~119 GB optimization is great. Hadn't seen that yet. That pops a 4x-Spark setup to 476 GB, which gives a lot more headroom for running something like GLM 5.2 at 4-bit, which isn't completely terrible.

I should have prefaced my post - I almost bought four Sparks a couple months ago, but ultimately opted to buy two more RTX 6000 Pro Max-Q's.

It was a painful choice because the two 6000's were more expensive than four Sparks, and ultimately gave me only 384 GB VRAM.

It was even more painful when GLM 5.2 was released, and a 4x Spark setup could run it at a decent quant, but 4x 6000's cannot with any headroom.

But the 6k's absolutely destroy the Sparks on prefill and inference speed. Model intelligence is compressing. The smaller VRAM pool will matter less over time than slower prefill/inference speed.

That is to say, I'm sure we'll end up with <500B parameter models that are Fable-level in the next 8 months or so. Performant quants of those will fit comfortably in 384 GB.

In your previous posting you said that for more than 2 Sparks you also need to buy a very expensive switch.

That is not really true. Once you have 2 fast Ethernet ports, like DGX Spark has, you can interconnect any number of systems without using a switch.

In the simplest case, you just daisy chain the systems and you configure in Linux the Ethernet interfaces as bridges.

For better performance, you can close the chain into a ring, with an extra cable. In this case the Linux configuration is more complex, because you must do IP-level routing, preferably with a routing protocol like OSPF, to be able to double the throughput between 2 systems, by using both paths through the ring.

The only advantage of a switch is a greater throughput when there are 4 or more systems, which happens only when all the interconnected systems are attempting to communicate simultaneously, in which case having only 2 paths through a ring will serialize some of the transmissions.

When only 2 systems attempt to communicate with each other, a ring has double throughput in comparison with a switch. You need 2 switches to match the throughput that a ring has, as long as it does not become congested. Or you could use one double-sized switch, with the ports partitioned between 2 VLANs, with each DGX Spark connected to 2 ports, in different VLANs. Both 2 switches or a double-sized switch would greatly increase the price.

For only 3 systems, a switch is useless, as it is worse than connecting the systems in a triangle and configuring the IP addresses for point-to-point links (the configuration as bridges is needed only for 4 or more daisy-chained systems). The configuration with 3 DGX Sparks seems optimal from the PoV of the performance per dollar ratio.

On each Spark you can bridge the two interfaces but you cannot bridge those bridges at layer two and it sounds from your post that you are confusing layer two and three (ARPA).

So if you have three Sparks, you have three two port switches. Take three two port switches and connect them in a ring and you have a collision storm. You have to use STP or similar to sort that out.

The ring is now collapsed to A-B-C (with no A-C) and to get from A to C you have to go via B which halves thoughput for those two paths: A-C and C-A.

Another "option" is to define two VLANs and use MSTP and layer three and hope the protocol that shuffles data can route packets effectively. So you get two sets of paths at layer two: A1-B1-C1-A1 and A2-B2-C2-A2 and you cut the two graphs at different points, so you get two disjoint graphs, for example: A1-B1-C1 and B2-C2-A2. However they will still overlap and it all falls apart, somewhat.

You can only link two Sparks together without a switch to get the full possible throughput. That is why Nvidia only offer a two pack option. Three or more requires a switch to avoid a path being used twice. Ethernet is not Token Ring! Mind you Tring was 16Mb/s - luxury when ethernet was 10Mb/s 8)

I've also had this discussion with folk when it comes to "hyper converged" virty systems: So we have three nodes with two NICs each for Ceph and we cable them up in a ring and it goes weird ... . At least there I'm only generally specifying 10G.

Now, do you really need the full 200Mb/s for clustering SParks? No idea yet, I've only got one and I've only got 40Gb/s QSFP+ to play with on my switches.

However I'm seeing value in the one box already at the moment and if that scales then I'll be buying more Sparks and a really funky switch when I need three Sparks and perhaps another switch down the line with some really fancy VLT or whatever wankery is the stacking du jour thing!

I agree with this. I have a dual rtx4090 machine, a 128gb m5 max mbp, and a dgx spark. RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic on the dual rtx4090 machine under vLLM absolutely slays at token speed. I'm frustrated with how hard it is to realize goodness on the dgx spark.

I have been running uConsoles with CM5 (2712 and 3588 with 16GB RAM) for 6 months as daily drivers.

They are ~$500* and present the same ARM problems/opportunities.

But they are completely silent (no fan, the case is the heat sink).

My 6600(3050) desktop from 2016(2024) with replaced SSD(2021)/RAM(2025) (they age like milk) now gets little use and M$ will soon sleep with the fishes.

*Hard to get now as the 3588 that has linux for uConsole is out of stock and the Raspberry one is rare and more expensive by the day.

Got 2 of these early on. They are wonderfully designed, unfortunately I've been caught in the "building things" trap for few months now and they have been relegated to being used as retro gaming/ computing learning boxes for my sons. They really don't appreciate it at all (yet)....but I can hope they will remember it in the future. Teaching them to get to terminal and run the emulator was great fun...reminded me of MSDoS and the hours of troubleshooting to run games with limited memory and drivers back in the day. I just worry that with LLMs the whole point of teaching them basic terminal/troubleshooting skills might be lost soon. We will see.

I would avoid the DGX Spark. For that price and its performance on running local models it is a complete scam. This tweet says it all [0]

[0] https://xcancel.com/petergostev/status/1978230978725507108

This is cool. Can I run the same fedora as the asahi Linux I have on my M2 Ultra and use the M2 to do builds and the DGX to launch kernels? That'd be the ideal fully arm at-home coding setup I think.

"Multi-token prediction gives a free speedup of up to 2x on many models" - at the expense of halving prompt processing speed

MTP has no effect on prompt processing.

Interesting - in my setup (llama.cpp rtx5090 qwen-3.6 27b) prompt processing with mtp is almost half vs non mtp. Sounds like I need to investigate what is wrong.

Maybe enabling MTP causes some weights to be displaced to host memory? MTP itself doesn't do anything during prefill so that should be exactly unchanged, decode will vary depending on settings but with 2-4 proposals depending on workload I've never seen an overall slowdown.

edit: I recommend building recent llama.cpp from source, I've been updating about once a week, as there has been a fair amount of work related to MTP recently. If you're running a lot of tool calling on Qwen you might also benefit from one of the bugfixed chat templates like the Froggeric version.

it works quite will for me in llama.cpp, but I get more like 20% to 40% speedup on tokens per second. I generally use

  spec-type = draft-mtp,ngram-mod
  spec-draft-n-max = 4

I have not observed any effect on prompt processing, which is usually an order of magnitude faster than generation on my spark.

Yes. Waiting for the Windows Version myseflf. RTX Spark Desktop.

I guess copilot needs all the help it can get?

It's interesting how many of these issues don't appear to be specific to the DGX Spark but to the standard "Nvidia GPUs suck on Linux" type of issues that afflict a lot of people.

it's actually very stable on x86_64 these days, even with optimus.

Is Nvidia better than Intel

This is something I'd do if I've had the disposable income lol

>Non-Steam games have a lower chance of working

Wonder if it's true for GOG games because they are usually installed in a neatly packaged folder without any bloat.

I'm genuinely disappointed with my Spark. I don't know how anyone can claim it performs decently with LLMs or diffusion models. Back when I worked in VFX in the early 2000s, we had a saying: "Render time is coffee time" and if you try to run this thing with a usable context size, you'll be drinking a lot of coffee. Most of the optimizations it relies on for inference simply aren't available for training, so it crawls like a snail on almost every model. An RTX 6000 Blackwell would have been the better investment for an AI enthusiasts and for general computing there are cheaper offerings.

If you bought it for inference you made a mistake. They aren't good at that. Use it to train models and experiment with ML. It's much better at that.

If you just want local inference buy a Mac.

If you bought early on, like I did, the Spark is probably worth double what you payed now. I think I paid $3,000 retail for mine and the last time I looked they were fetching close to $6k on ebay. I'm not sure if that's still the case, but you can buy a very nice Mac with $6k.

Mhm... maybe read my full comment?

Are you saying that it's slow for training? Sorry, your comment is confusingly worded to me.

I've not had any issues in that regard, but I'm working with LLM's not training diffusion models. Are you following one of the Nvidia provided recipes or inventing something on your own? The last time I looked into it they benchmarked very well, but we both know that doesn't always mean much.

Yes it is slow for training/fine-tuning decently sized LLMs and unusable for diffusion models. What are you training were you get usable speeds?

The DGX Spark hits a sweet spot for me where it can simultaneously work for general computing and run local LLM inference fast enough for some hobbyist dabbling. The RTX 6000 Pro Blackwell is more than twice the price (it has increased quite a bit recently from $8000 to $11600), not to mention the "rest of the PC" needed to get it working, so it's not really a fair comparison. Compared to other 128 GB unified memory devices like the Mac Studio and the Strix Halo, the DGX Spark fairly priced in my opinion.

I have a AMD Strix Halo box I use for similar dabbling. It definitely wasn't an "out of the box" experience, fiddling around with kernel versions and ROCm installs. These days I mostly wind up using the Vulkan build of llama.cpp for inference.

I have access to both a RTX 5090 PC with 64gb of ram and a Spark 128gb, the performance of the Spark has been highly disappointing.

I prefer 1000x the RTX one, even with 64gb of ram.

Reply on news.ycombinator.com