Comment by SwellJoe | Hacker News Reader

Comment by SwellJoe | Hacker News Reader

Parent Full thread

The unusual bootloader and custom hardware makes modifying and upgrading the OS a challenge. I work on robots that need a custom OS image loaded on the machine. The Nvidia Ubuntu version makes that a huge pain in the ass. They've got binary-only drivers that have to be there, the install/upgrade process is finicky and prone to failure (always recoverable, so far, but all the techs in production I work with have a hard time with it and have to be walked through it, as it's just so alien to folks who work with regular PCs most of the time).

Let's just say if I had my druthers, I would not choose Ubuntu, and I really wouldn't choose the Nvidia/ARM spin of Ubuntu. The Strix Halo has the benefit of being entirely a normal x86_64 PC that happens to also have a big chunk of unified memory. You can put pretty much anything on it. Any Linux, regular Windows, probably even a BSD (though good luck getting AI stuff working there).

But, as I said, if you're spending $4000 exclusively for inference and AI workloads, you might as well get the Nvidia-based unit. It is better for that.

No the Spark uses UEFI + ACPI

Regular Ubuntu or Fedora ISOs do work out of the box

Just needs the Nvidia GPU driver install afterwards

(And the realtek 10gbe module oot, or blocklist if not used)

And for the Jetsons btw still uses device tree with custom kernels but at least it's UEFI from Orin onwards.

For Jetson Orin and later:

You can download an ISO from https://developer.nvidia.com/embedded/jetpack/downloads and that'll work. Other distributions are still a bit of a mess though but Yocto is supported now.

Ah, that makes all the difference. Being normal has a lot of value. I mostly take it back. I still don't like the custom Ubuntu Nvidia ships, but if the system is normal any Linux distro for ARM works, it becomes an entirely viable thing.

Nvidia didn't make a great out-of-box experience for sure, one of the first things I do on my Spark is set init 3 and disable bluetooth. Docker was working on one, not the second. Many typical local tools used in Ai were not installed (ollama,vllm,llama.cpp... all MIA)

I mean, I never expected to have pre-packaged versions of any of those things that wouldn't just fall out of sync with upstream immediately anyways?

I basically treated it as a stock Ubuntu machine but on Aarch64, and ignored the stuff they installed. And then went looking for docker images for vllm that made sense and were up to date and hopefully tuned for NVFP4 on the hardware and...

... that was more the disappointing part.

The Spark does a system update on first boot, so they would have updated had they been installed

I'm running vllm + GoModel on k3s, works like a charm. Even wrote some CUE last night to generate the values file for my Helm chart. It calculates the GPU percentage for me from the memGB I assign to a model.

Yeah, the stuff they preinstall is crazy. It's a ~16GB OS image, which is a huge pain in the ass (in addition to all the other things that are a pain in the ass about Nvidia's OS situation); I can't ship their 16GB pile of crap plus our custom 3GB of stuff on my robots, there's no reasonable way to do OTA upgrades with a pile of shit like that. It's just an unpleasant experience all around. What I always want is a nice clean basic install, and then my custom packages of everything I need goes in. Especially for embedded cases, like the Jetson line is allegedly intended for, their whole approach is just stupid, and they don't provide any documentation for how to not use their stupid thing.

Still, I'd rather have the blazing prefill speed of the Nvidia over my Strix Halo, but I wasn't willing to spend nearly twice as much for it (when I bought, the Strix Halo was $2100 and the GX10 was going for $3700, I think). Now that the difference between the two is much smaller, there's no reason to get the AMD.

The $2100 vs $3700 price difference really came down to the ConnectX hardware in the thing, I guess. But the price gap narrowed over time, now they both suck.

I am not sure why NVIDIA couldn't have released a cheaper version of the thing with just standard Ethernet on it and leave it at that. Hardly anybody can afford two or more of them to cluster via ConnectX anyways.

Reply on news.ycombinator.com