Comment by gerdesj | Hacker News Reader

Comment by gerdesj | Hacker News Reader

Parent Full thread

Actually, its ~119GB usable (I have one). You can shutdown a lot of unneeded services if you only use it remote which will trim a lot more fat. You can enable the RDP service if you don't want to sit in front of it but get a desktop interface.

The "shitty" network is 10Gb/s and wifi7! You get a twin QSFP28DDlol+++ (I jest) that each run at 200Gb/s - not for the casual home user but handy at work, although I "only" have 40Gb/s on my switches sigh. With and no switch two you can do a three node cluster with some careful networking. If you want to do more then a switch is needed and it will need to be pretty funky! That said you could wire them up in a circle and use VLANs and MSTP and accept less than 200Gb/s per link. You'll probably need Openvswitch and a lie down afterwards.

I'm not a fan of the Gnome desktop but it works well enough and I think the Nvidia customised Ubuntu is well thought out. You get all the complicated NVidia extras pre-installed, along with docker (full fat, not the Ubuntu one) for a fairly quick start. It includes Ubuntu Pro which is free for five systems anyway but its nice to see it pre-installed.

You can run quite decent models on this thing see: https://spark-arena.com/ Also see "DS4".

We blew abut £4000 on one and it will pay for itself in a few months. I tried pricing up an Apple thingie and the Store wouldn't offer me more than 96Gb of RAM and a delivery date in Q3 at the earliest. Our Spark rocked up next day. They seem to come in 1TB or 4TB SSD variants. 1TB is enough for me and saves a lot of cash - keep an eye on your model downloads and ruthlessly delete old experiments. docker system prune.

We went for the Asus variant that has active cooling and I stuck it in the ceiling cable tray over our computer room racks. It sits on 1½" stainless steel mesh with lots of clearance in an actively cooled environment.

That ~119 GB optimization is great. Hadn't seen that yet. That pops a 4x-Spark setup to 476 GB, which gives a lot more headroom for running something like GLM 5.2 at 4-bit, which isn't completely terrible.

I should have prefaced my post - I almost bought four Sparks a couple months ago, but ultimately opted to buy two more RTX 6000 Pro Max-Q's.

It was a painful choice because the two 6000's were more expensive than four Sparks, and ultimately gave me only 384 GB VRAM.

It was even more painful when GLM 5.2 was released, and a 4x Spark setup could run it at a decent quant, but 4x 6000's cannot with any headroom.

But the 6k's absolutely destroy the Sparks on prefill and inference speed. Model intelligence is compressing. The smaller VRAM pool will matter less over time than slower prefill/inference speed.

That is to say, I'm sure we'll end up with <500B parameter models that are Fable-level in the next 8 months or so. Performant quants of those will fit comfortably in 384 GB.

In your previous posting you said that for more than 2 Sparks you also need to buy a very expensive switch.

That is not really true. Once you have 2 fast Ethernet ports, like DGX Spark has, you can interconnect any number of systems without using a switch.

In the simplest case, you just daisy chain the systems and you configure in Linux the Ethernet interfaces as bridges.

For better performance, you can close the chain into a ring, with an extra cable. In this case the Linux configuration is more complex, because you must do IP-level routing, preferably with a routing protocol like OSPF, to be able to double the throughput between 2 systems, by using both paths through the ring.

The only advantage of a switch is a greater throughput when there are 4 or more systems, which happens only when all the interconnected systems are attempting to communicate simultaneously, in which case having only 2 paths through a ring will serialize some of the transmissions.

When only 2 systems attempt to communicate with each other, a ring has double throughput in comparison with a switch. You need 2 switches to match the throughput that a ring has, as long as it does not become congested. Or you could use one double-sized switch, with the ports partitioned between 2 VLANs, with each DGX Spark connected to 2 ports, in different VLANs. Both 2 switches or a double-sized switch would greatly increase the price.

For only 3 systems, a switch is useless, as it is worse than connecting the systems in a triangle and configuring the IP addresses for point-to-point links (the configuration as bridges is needed only for 4 or more daisy-chained systems). The configuration with 3 DGX Sparks seems optimal from the PoV of the performance per dollar ratio.

On each Spark you can bridge the two interfaces but you cannot bridge those bridges at layer two and it sounds from your post that you are confusing layer two and three (ARPA).

So if you have three Sparks, you have three two port switches. Take three two port switches and connect them in a ring and you have a collision storm. You have to use STP or similar to sort that out.

The ring is now collapsed to A-B-C (with no A-C) and to get from A to C you have to go via B which halves thoughput for those two paths: A-C and C-A.

Another "option" is to define two VLANs and use MSTP and layer three and hope the protocol that shuffles data can route packets effectively. So you get two sets of paths at layer two: A1-B1-C1-A1 and A2-B2-C2-A2 and you cut the two graphs at different points, so you get two disjoint graphs, for example: A1-B1-C1 and B2-C2-A2. However they will still overlap and it all falls apart, somewhat.

You can only link two Sparks together without a switch to get the full possible throughput. That is why Nvidia only offer a two pack option. Three or more requires a switch to avoid a path being used twice. Ethernet is not Token Ring! Mind you Tring was 16Mb/s - luxury when ethernet was 10Mb/s 8)

I've also had this discussion with folk when it comes to "hyper converged" virty systems: So we have three nodes with two NICs each for Ceph and we cable them up in a ring and it goes weird ... . At least there I'm only generally specifying 10G.

Now, do you really need the full 200Mb/s for clustering SParks? No idea yet, I've only got one and I've only got 40Gb/s QSFP+ to play with on my switches.

However I'm seeing value in the one box already at the moment and if that scales then I'll be buying more Sparks and a really funky switch when I need three Sparks and perhaps another switch down the line with some really fancy VLT or whatever wankery is the stacking du jour thing!

Reply on news.ycombinator.com