Will We Ever See InfiniBand in Desktop Computers? | Hacker News Reader

Will We Ever See InfiniBand in Desktop Computers?

clusterdesign.org

25 pointskungfudoi22 comments

Why switch to infiniband? Ethernet can go just as fast (not as good of latency) and the cables are cheap and ubiquitous. The only thing stopping higher speeds is cost. I have been watching 10GbE PCI-E cards for the last year or two and they have dramatically dropped in price ($200 from newegg). You can get server motherboards with 2 integrated 10GbE ports for $500 as well. 10GbE does run over cat6 by the way.

I think for general-purpose workloads, it's not relevant. However, some of the IB-specific capabilities like RDMA are extremely interesting for clustering use cases. You can dramatically reduce the overhead of moving data around between nodes by bypassing the kernel entirely.

I understand that sort of use case, but this article is asking about desktop computers. It almost seems like the author only sees it used in high performance situations so clearly it is better for all situations.

thunderbolt and the predecessor firewire both have rdma.

There is no practical or commercial motivation for IB on consumer devices. Sure one may make a case that a home server or a media center can use IB to stream photos and movies to other devices. At this point, we have to stop and think like an average consumer and not an IT professional. And an average consumer prefers no wires forget running copper or fiber. And what benefit would it offer over wireless even if they were to run the cable? None from the consumer perspective. You can stream HD over wireless without any problems.

Sure, we all prefer wired over wireless when available. But not in home, and not by an average consumer.

Intel will integrate IB with their future processors, but they will not be consumer based SKUs.

There are niches -- some people (interested parties for sure) even advocate for fiber to the desktop as a cost savings: http://www.thefoa.org/tech/allfiber.htm

But for regular consumers, wireless is the driver. Power users will care about cat 6. The rest is just niche.

10GBaseT is cheap. Look at Arista networks. This idea is just not well reasoned. Ethernet had all the "stuff" needed to make it work when you plug it in. IB requires you to do a lot of configuration on the IB switch side, etc.

You can do InfiniBand on the desktop, but it necessarily doesn't give the performance you might be expecting. I think I read about this on HN originally...

http://www.davidhunt.ie/infiniband-at-home-10gb-networking-o...

Edit: From the edit on the follow-up post about getting it working on Ubuntu:

>Note: iperf maxed out at 1.2 gbps, and on the current linux install, I couldnt get netperf client working at all. netserver would work, but only showed a throughput of 25mbps from a Win7 client. HOWEVER, when I set up the raid with 6 old 160G drives, the “hdparm -t /dev/md0p1″ showed 250MB/sec reads, and I got the same from the Win7 machine using samba across the infiniband fabric. This seems to indicate that iperf and netperf are completely unreliable for testing this type of connection. Bear in mind though that I did have netperf running on the previous ubuntu install, but that installation was so messy I don’t know what drivers and user-space software was running. I reckon it’s the kind of think that may be fixed in the stock Ubuntu install in the near future. For the moment, just go with real-world testing, i.e. copying large files from ramdisk to ramdisk, for example.

In visual effects, there is need to transfer around massive data sets. WE have found that 1 Gbps Ethernet just doens't cut it, but right now 10 Gbps Ethernet solution are just too expensive for now, or require specialized cables (fiber optics).

http://en.wikipedia.org/wiki/10-gigabit_Ethernet

I would really like those prices to come down and also to use just regular CAT cables.

Right now it seems that the switch to faster than 1 Gbps networking is just too costly and not easily incremental (because of the switch to optical generally.)

PS. There is 10GBASE-T but I think there is zero adoption or promotion of this substandard: http://en.wikipedia.org/wiki/10-gigabit_Ethernet#10GBASE-T

10GBASE-T isn't popular because your cable runs are cut short.

Costs have come down a lot on fiber. A short range SFP is like $120 or so, and short range cables are like $3 per meter.

What a lot of people do as a stopgap is binding ports together, using something like LACP. There's trade-offs of course. A 4 port 1000BASE-T card could be used to boost a few important servers.

10GBase-T also entails more power consumption per port and significant latency over other 10GbE standards.

10GBASE-T is pretty well established for top-of-rack datacenter switches. Intel's X520 chipset is pretty common and any major switch vendor will happily sell you a 24x or 48x 10-gig switch (HP's JG336A for example).

Twinax is arguably better if you're building a rack from scratch, they're lower power, but if you're upgrading something existing it's hard to justify rewiring everything. Also twinax cables are expensive and annoying to work with.

I don't mean to be cynical, but other than for niche clusters in the home, wouldn't such technology be hobbled instantly by incoming bandwidth?

Even in Germany, where the author lives, the top bandwidth appears to be about 50 Mb/s:

http://www.pcmag.com/article2/0,2817,2457284,00.asp

Which although it is still much better than the states:

http://ispspeedindex.netflix.com/results/usa/graph

They're both well below 1 Gb/s, no?

I don't think the use case envisioned here is for home computers with InfiniBand, it's for work stations in corporate environments. Transferring large data sets to local servers is quite common, and can take a (relatively) long time even over Gigabit Ethernet.

Corporations can afford to buy pricy network hardware for the people who need it (or they can skimp on engineering resources and congratulate themselves over and over again over how thrifty they're being).

I read desktop to mostly mean consumer, perhaps erroneously?

It's mostly for moving things around your internal network. I would love to have 10Gb to my NAS, because then my backups would take seconds instead of minutes. (First world problems, I know).

Interesting. I think it's possible that Infiniband will one day work its way to desktop computers, but there are a lot of players in the game that have to agree to make it work and make it popular.

Would you want every workstation to have direct access to the cluster - ok I get for loading data it might be useful but IN that case surely you would have special purpose nodes for that

How would Infiniband be so much faster than Ethernet over the same cables? And would a 5x faster Inifiniband be less than 5x as expensive?

Going off of the [Wikipedia page](http://en.wikipedia.org/wiki/Infiniband), I think it's due to the "switched fabric topology" of InfiniBand.

Ethernet is mostly switched now (when was the last time you used a hub?), and the topology doesn't change the line speed.

No.

Reply on news.ycombinator.com