Hacker News Reader

Comment by paulsutter | Hacker News Reader

Is there any reason a networking switch couldn't be built with Thunderbolt ports (20Gb/s)? Or for that matter, USB 3.0 (5Gb/s)? Seems a cheaper way to get higher speed interconnects for server clusters than using Ethernet. Sorry if its a naive question, just curious.

EDIT: This would be for new installs, of course. You'd use a motherboard with only a Thunderbolt or USB 3.0 port.

EDIT: I'd love to hear the "more complicated reasons", as your other objections dont apply. In a compute cluster, you have short (within-rack) cable lengths, and the owner of the switch is the owner of the cluster so security is not an issue. Cost is a super big deal in a cluster. More expensive interconnect means buying fewer servers, and that's just dumb.

View on HN

The more complicated reasons aside for the moment, thunderbolt is more or less like an external PCI express connector (with some added goodness). As a result you can access a lot of things via thunderbolt you probably wouldn't want outwardly accessible (imagine a network where anybody could do a DMA on demand). Then there are a number of other things, for example the protocol and phy not being designed for 100 meter cable runs like Ethernet, no low level networking support, the fact we have a reasonable and ubiquitous interconnect for networks already.

There are alternatives for super high speed interconnects within a data center though (think infiniband and cousins) or high bandwidth options for longer distances (like fiber) etc.

Infiniband and 10Gb ethernet are both ludicrously expensive steps up from gigabit ethernet which is what makes people look for alternatives everywhere.

Back in the day there were people doing IP over (P)ATA and SCSI for similar reasons... Cable length is not necessarily a big deal - for a lot of applications the appeal would be to use it as an interconnect between servers in the same rack.

10GE and Infiniband are actually not that expensive for HBAs now. Just expensive for switches, but not that expensive, and would be just as expensive for any other non-blocking switch with that level of bandwidth. Juniper EX4500 is probably the cheapest right now, at around $20k for 40 port.

$500/HBA and $500/switch port is cheaper than GE was at the same point in its adoption cycle, I think. Might even be half that with discounts. Aggregated GE is still cheaper and makes more sense for servers in most cases, though.

The obvious response is that you'd have to swap out all the Ethernet hardware in order to get the speed advantage, which gets a bit expensive.

The other half is software, maybe, but I honestly doubt that is actually the problem here. We've been encapsulating protocols in other protocols for so long that that problem is essentially solved in all but the weirdest, most obscure cases, and this wouldn't be that weird or obscure.

Reply on news.ycombinator.com