Intel wants to kill the traditional server rack with 100Gbps links
arstechnica.com
arstechnica.com
- High speed copper physical layers are cheap - Thunderbolt today has 40Gbps PHYs, and Infiniband/10GBE both extremely inexpensive compared to a few years ago.
- Their chips are getting smaller and more power efficient, and the Atom finally has an ECC part.
- As clockspeed is not longer hockeysticking up, people are transitioning in droves to multi-system parallel task model, which favors lots of medium speed, efficient cores.
In short, in the near future general purpose computing is going to look more like HPC clusters, because the software side of things has, in most cases, caught up.
I'd assume that Intel's long term trajectory is to have CPU/cache/memory/flash all on one system-on-a-chip "module" that is the smallest replaceable subunit. Larger dedicated storage is delivered over the network interconnect in a tiered architecture.
Now 10 us vs 10.3 us might not sound like much but it's still 3% slower. And it get's worse when you look at high end DRAM based SSD's which can be faster than 10 us.
According to Wikipedia, it depends on the insulation:
== SNIP ==
Propagation speed is affected by insulation, so that in an unshielded copper conductor ranges 95 to 97% that of the speed of light, while in a typical coaxial cable it is about 66% of the speed of light.
== SNIP ==
For LMR-400 (a very common cable for Ham Radio) - the Velocity Factor is about 85% according to http://www.febo.com/reference/cable_data.html
The net-net is, that there are opportunities to get closer to the speed of light if latency is really, really important.
66% turns out to be a surprisingly consistent approximation for both copper and fiber.
A few clock cycle might not matter for bulk storage, but Intel is also talking about separating main memory from individual processors. There individual clock cycles do matter. Witness the rise of low latency premium RAM.
From what I understand, the article is about two things:
1. Hot swappable components.
Oversimplified, but imagine being able to add / remove CPU and RAM to your server like you can with disk space and USB thumb drives.
2. Shareable resources / load balancing.
If one server is using all its CPU and another is using all its RAM, they can now use each other's resources.
You know how AWS is on-demand, scalable computing? This is the same thing but at a hardware level with CPU, RAM, disk, and network resources.
[0]: http://static.ddmcdn.com/gif/computer-memory-pyramid.gif
They're basically taking the backplane of a motherboard and spreading it across a whole rack instead of inside a single server. Especially memory would take a latency hit if all you do is extend the "cable" length between the CPU and memory, which is why they also need a much faster interconnect.
Or pretty much any decently spec'ed new RAID array. Most of our "recent" arrays are in the 300MB/sec to 500MB/sec range and were not particularly expensive.
But you're probably right - while it'd be nice to be able to hang those arrays off iSCSI or export network filesystems that can reach those kind of speeds, it's rare-ish to need it. Saturating GbE in a way that doesn't make it just as easy to "just" bond a couple of interfaces together or spread your IO load over a couple extra servers is a pretty special case still.
I'd love to be able to justify 10GbE in our network, but I can't until it costs almost as little as GbE.
> and nothing remotely consumer-level can saturate 10GbE yet.
Not much consumer level even saturates Fast Ethernet. Especially given how much consumer equipment today still hangs off 54Mbps or below wifi... I don't think that's really an argument - most 1GbE equipment likely still goes to corporate networks.
At home, I'm running an 8-port SATA RAID controller (but with ZFS, so not using the RAID firmware). I'd say it could saturate 10GbE. But it's true I'm not the average consumer.
I always hate to get big files from our NAS at 7-8MB/s when a proper GBit device can achieve 10x that. That isnt even shared, if 2 or more people try to get files from the thing, you can basically forget it.
Turns out the VoIP phones were now acting as switches for all the Desktops, and my Desktop had been plugged into the phone, which, you guessed it - only had a 100 Mbit interface.
Moving my desktop back to my (still lit) Gigabit port returned me to my regular speedy connection.
Gigabit makes a big difference. I'd hate to throw around multi-100 megabyte files on 100 megabit network connections.
Enough things are CPU/memory/etc. bound that the "cheap" building block of a 1-2CPU server with 4GE, RAM, and some local disk is still more appropriate than buying a Xeon E7 with 10GE HBAs, in most cases. There might be an exception if you have per-host vs. per-core licensing for expensive stuff, or other artificial complaints, or a specific component (database?) which doesn't horizontally scale.
I predict $5k 40-port 10GE switches and commodity server on-board 10GE NICs in a couple years, though. Although at that point, you need something crazy to uplink the switches. 40GE is emerging, or you could use a non-ethernet option. SANs are the big application for 10GE now since you can comfortably fit all the clients and servers on a standard 10GE switch and don't need to uplink most of the traffic. Part of the issue with the higher speed ethernets is lack of a copper cabling option, particularly one which works with existing cable plants. (less of a concern within racks).
Now people are realizing they can build a server from a bunch of parts connected by fast fabric, and you can pull/place items in the fabric and use them immediately, or take them apart. The MULTICS machine was actually carved into two pieces live, every night, to run two instances, then re-merged in the morning(!)
Hardware virtualization - and component aggregation - were always great ideas, and now the technology exists to deploy it at the middle-tier commodity server level.
To be clear, I like mainframes. They make certain problems much easier. I worked in the company that maintains the world's largest DB2 installation and it was amazing and terrifying.
Instead of a dumb rack -- four posts and screw holes in the right places -- you get a blade chassis 72U high. Centralized but redundant power supplies with integrated monitoring and per-feed software control. A built-in KVM. Large slow(er) fans that push more air more efficiently and more quietly, or perhaps a liquid cooling system that provides a clean, standardized disconnect for each component. Assignable resources -- instead of running virtual machines, you run re-configurable real machines, where you start by selecting a number of processors with associated RAM and add in storage.
On the one hand, this will have less in common with consumer hardware, so economies of scale will not be shared. On the other hand, server hardware is already substantially different from high-end consumer hardware, so it's probably not that big a deal.
We discussed the challenges of various signaling that would prevent companies from being willing to participate. But this idea is really old.
Probably a nice idea though
Just like the way Cisco, et. al. are co-opting SDN...http://www.lightreading.com/blog/software-defined-networking...
But what's actually happening is that Intel is building a fabric interconnect which can serve data at DRAM-bandwidth-or-higher speeds. And that's certainly something that hasn't been done in the modern world. The headline might lead you to believe it's the architectural idea that's the new thing here, but it's really the pure technology that's the interesting bit.
Yes, that is exactly what I am talking about. It was perfectly normal to purchase systems in the 90s that worked that way. A cabinet powered by a single power node, and you plugged in CPU nodes, memory nodes, I/O nodes, etc.
Now (apparently) there's a new interconnect that can do the job. That's news, not "same old boring stuff".
The rest of your post is simply factually incorrect. People did not stop doing that, such systems continued to exist through the 90s and 2000s. They still exist right now. They became less common, but that had nothing to do with "the interconnect just wasn't fast enough". It was because intel CPUs became the fastest available, and racks full of small intel based systems were (and are) massively cheaper and entirely sufficient for the vast majority of uses. The speed of interconnect fabric kept up just fine.
The SGI shared-memory big iron machines (Origin & Altix) are more recent and can have memory-only nodes. The most recent NUMA Altix was launched in 2009, I'm not sure if the later machines managed to keep the Origin 2k era goal of having as much remote memory bandwidth as local.