Hacker News Reader

Comment by nickdothutton | Hacker News Reader

I see we’re doing this again. I last worked on whole-DC designs around 1999 with Sun, Intel, and of course… Mellanox since we were to use RDMA/InfiniBand. The plan was to embed InfiniBand semiconductors on disks, RAM (individual sticks or trays of sticks) and CPUs. There was to be an orchestration/provisioning layer allowing you to reserve a composable machine (CPU(s), RAM, storage, other IO, etc) at will, and some kind of supervisory layer was to ensure that these things were as local as possible to one-another in any given rack, aisle, floor.

View on HN

So you were pretty much working on a more hardware based predecessor of kubernetes?

Why did it not work out?

It wasn’t written in Go </snark>

I’d also love to hear more about the concept.

Go peaked in 2017. Rust is the new hotness. 3 years from now, it will be something else.

> Why did it not work out?

Software won, it seems.

kinda, It was cheaper to use fibrechannel/ethernet and plain old sparc/intel machines

This was about the time that grid engines promised mainframe computing without the cost. Only that it required people to understand how to parallelise their workloads. That and modify programmes to run in on many machines at once.

> Why did it not work out?

1) cost

2) software support

3) cost

Some big problems that jump out:

* Infiniband hardware didn't get cheaper as fast as RAM + disk

* Infiniband hardware didn't get faster as quickly as locally-attached RAM + disk

I think a few things combined to torpedo it. Firstly we were hit with the dotcom crash. Secondly Sun’s future rapidly looked less bright, and they were the main server partner in the gang. Thirdly there wasn’t (yet) the “Datacenter OS/Orchestration” layer. It didn’t exist and was the part that would allow us (I think worlds largest ISP and shared/dedicated hosting provider at the time) to carve up the data center on-demand. Remember this is ‘99 so there was no VMware server for another ~2 years. Finally, x86 rack servers got “good enough” and that low cost Pee-Cee architecture took over (with more than a little help from Linux). Yes it was hard to orchestrate. Yes it lead to sprawl (before even VM sprawl) but the market didn’t care.

Infiniband was mostly displaced by 10, 40, & 100GbE due to costs per port.

With costs so high, adoption is reduced.

The vendors would've been smart to give away one part of it.

Our prod environment doesn't use it because millions of servers would be cost-prohibitive. Corp uses it here and there for experimental clusters.

Reply on news.ycombinator.com