Introducing “6-pack”: the first open hardware modular switch | Hacker News Reader

Introducing “6-pack”: the first open hardware modular switch

code.facebook.com

251 pointsm0nastic95 comments

These big switch boxes typically end up being about thermal management and this box looks like thermal design was an afterthought. Also I'm not sure the power entry design is really all that smart. Generally speaking if you're in the market for a 640Gb/s switch (or 3.8Tb/s switch), does your data center really not have access to 48V power? The AC/DC conversion wastes power and space.

The QSFP's are "spaced for optimal airflow." However this spacing seems to neglect cooling the QSFPs themselves. Belly-to-belly mounting of modules is usually the most thermally challenging way to arrange them. The heat dissipated by the QSFP's is generally directed towards the top of the module. By placing open air channels between modules, they have effectively ensured that little to no air flows over the QSFP heat sinks (which is not shown). So there is probably a limitation on which reach codes are supported. My guess is that because of the thermal limitations of this design, it's not truly non-blocking in all reach configurations.

does your data center really not have access to 48V power?

No, they really don't. I've never heard of 48V outside of telco.

I've been to lots of them all over the world, names you'd recognize, and they've all had 48V.

Facebook's Open Rack project is based around 48v.

Yes they do. My old employer TeleCity installed 48v as standard, for their telco customers.

It's pretty common here (nl), you have to ask for it but it is usually available. I'd advise against it because you're deviating from the bulk but if you need it you can usually get it, especially at the larger colocation facilities.

All the Pro's (Cisco Juniper etc) offer 48v Kit as server makers targeting the carier grade market.

I have 48v in both my home rack, and the offices rack.

This is true, however: 48V circuits are generally priced 'per-amp' just like 120V or 240V circuits. That translates into much higher cost-per-watt and with the efficiency of AC switching supplies being well over 90% you're probably better off from an economics point of view of getting the highest voltage supply that you can get and then to step down in your rack mounted devices.

I very much want to transition to open hardware models for networking gear - for both my own home use and for rsync.net.

However, at no single site (especially my home) do I need a 6U chassis full of switch ports.

Is there a 1U version of this on the horizon ?

Facebook's Wedge 1U switch is what this builds on. https://code.facebook.com/posts/681382905244727/introducing-...

Is anyone building a copy of that to sell to other people besides Facebook?

Similar to how you can buy a Backblaze Pod ( https://www.backblaze.com/blog/backblaze-storage-pod-4/ ) from http://www.45drives.com/

I haven't seen anyone selling Wedge specifically, but it's not that different from what you can get from Quanta, Edge-Core, or Penguin Computing.

Apparently juniper is with their OCX1100[1] model.

[1]: http://forums.juniper.net/t5/Data-Center-Technologists/Junip...

All of these things are based on the Broadcom Trident II chip. If you want it cheap, don't go to juniper - get one from Quanta. You can have a 32x40G switch for $6,000.

thanks, good to know!

On a similar note, I just recently became aware of Cumulus[1] Debian based switches (but the good bits are closed source) from among others edge-core[2] (via a presentation by PaaS-provider http://zetta.io) -- eg:

http://whiteboxswitch.com/products/edge-core-as4600-54t

[1] http://cumulusnetworks.com/support/faq/

[2] http://www.edge-core.com/

We've been loving Cumulus + Quanta for 10Gb and 40Gb, in that it's more manageable than Cisco (for our environment) and a fraction of the cost. We end up using it at 1Gb too, but it's just a price match there, instead of a win.

To clarify, the only part that is closed is "switchd", which is a userspace program that watches the kernel data structures (route tables, neighbor tables, bridges, ports, vlans, etc) and programs the hardware to match. It links against proprietary silicon vendor SDKs, and programs registers whose description were given to us under NDA.

Without this part, everything works the same, but is of course not hardware accelerated. So the 100% open source parts of Cumulus Linux would still make a great Network OS for a router/switch VM.

We don't yet have an official VM version, but that is something we will have in the future.

- nolan co-founder/CTO Cumulus Networks

What is the flexibility with "open" switches? To get linerate switching, I'm guessing you're still limited by the hardware? Is the benefit that you can more easily setup routing tables (instead of depending on the switch vendor's capabilities), vlans, etc. just by creating them in userspace then pushing them over to the hardware part?

Or can you actually get fairly low level, like implementing your own algorithms for channel bonding? A while back I wanted to do some L7 inspection, but could only get like 10G per server, and we had 40G coming in. EtherChannel didn't acceptably balance out the traffic. Doing so would have required dealing with one of the network processor vendors and all that mess. Would an open switch platform make this a straightforward exercise?

You are limited by the hardware, and what our code supports programming into it.

The big advantages are reusing config management tools like puppet/chef/ansible/etc, and monitoring tools like collectd/graphite/nagios/etc.

Also, it is super easy to run services on the switches. For example, you can easily run isc-dhcpd on each ToR, instead of DHCP relaying back to one mega DHCP server. Distributing services like this scales better, and reduces the blast radius of service failure.

I've been experimenting with the idea of a transparent caching TFTP proxy server running on the top of rack switch, to make PXE scale better to large clusters.

The important thing is that anyone who has the know-how to write a transparent caching TFTP proxy server for Linux can just go ahead and do that on a Cumulus Linux switch! You don't need to come to us and convince us that it is a good idea and then wait for us to actually implement it. Compare that to asking for features from a traditional switch vendor...

- nolan co-founder/CTO Cumulus

I'd imagine folks like Quanta or Hyve Solutions will be selling something like this soon.

You probably don't need 16 40Gbps channels at home, either.

Heh maybe not need, but want? I definitely want.

Looks like a 40 GbE NIC is just $500: http://www.amazon.com/MELLANOX-CONNECTX-3-56GBE-1PORT-MCX313...

Just need an SSD RAID to feed it...

Less than $500 actually, you can get them for about $350.

While I wasn't aware you could get them that cheap, I think I'd still look into second hand infiniband for home use. Allows the use of copper for short (pc-to-pc, point-to-point) distances. While optic is absolutely cool, as far as I can tell a single 1m patch cable will cost ~80 USD -- quite a lot considering the cost of the NIC... And even in IBM's sales brochure for their optical switch, infiband comes out a little ahead:

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=S...

40GbE uses approximately the same copper cables as Infiniband. There's probably more used IB equipment floating around than used 40GbE, but otherwise I'd go with Ethernet.

Ah, of course. I was misled by wikipedia[1], but it should've been obvious that the same connection could be used for both inifinband and ethernet. So, indeed, there are copper interconnects:

http://www.amazon.co.uk/Mellanox-3-3ft-QSFP-QSFP-Passive-Cop...

Actually seems the price is finally coming down a bit (compared to what I remember these used to cost, years ago -- but maybe I've just upped my budget ;-).

[1] http://en.wikipedia.org/wiki/QSFP

"The Quad Small Form-factor Pluggable (QSFP) is a compact, hot-pluggable transceiver used for data communications applications. It interfaces networking hardware to a fiber optic cable."

But it also interfaces hardware to copper cables, as I gather.

The cable you're looking for is called a direct connect cable. "Fake" optical modules on both sides, plugs it together with a permanently wired copper cable.

DAC is a common term for this. Direct Attached Copper.

It is a SFP+ plug on each end (but without all the optical magic), connected with twin-ax cable, which is like coax but with 2 signal paths, one for each direction.

http://onie.org/

For example:

http://www.edge-core.com/ProdDtl.asp?sno=425&AS4600-54T

http://www.edge-core.com/ProdDtl.asp?sno=423&AS5600-52X

Basically anything on here: http://cumulusnetworks.com/support/linux-hardware-compatibil...

Some easy ways to buy are: https://bm-switch.com/ http://whiteboxswitch.com/ http://ipfabrics.net/ http://www.penguincomputing.com/products/network-switches/

- nolan co-founder/CTO Cumulus

Are there any that are, say, 48 port GigE? 10G and 40G are a bit much for home use...

10G and 40G are a bit much for my office, even.

There are 1 gig 48 port switches on Cumulus's HCL, they're around $2500 and $700/yr for Cumulus support. Unlike Cisco or Juniper, you must maintain support in order to continue using the product (legally). I can't imagine getting enough utility from a switch for this to make sense for home use.

It is substantially better pricing than list for comparable Juniper / Cisco / etc. equipment. However, Cumulus has no truly low end 1 gig switch (single power supply, limited L3 capability), and you can absolutely negotiate Juniper / Cisco / etc. down to be close to or even below the pricing of the Cumulus solution. That is tougher for 10 gig or 40 gig equipment, which is where the value proposition kicks in for Cumulus.

Maybe Cumulus's approach is enough of a value add for it to make sense to pay a premium, but everyone I talk to is interested in cost savings first and better manageability a distant second. You'll still have to have Juniper / Cisco / etc. in your life to an extent, Cumulus doesn't do routers and they don't have a full range of switch models.

Stay tuned for low-end 1G platforms running Cumulus Linux.

- nolan

Hey Nolan, I love Cumulus' approach and I bet you guys will go very far!

I thought about adopting Cumulus for a large telco project, but for the moment it doesn't seem a good fit as we do lots of openflow, L3 and custom application development.

Actually, have a look at it. It's also blue (maybe we should sue facebook :D): http://ripe69.ripe.net/archives/video/177/

I'll take a look, thanks for the link.

We don't do OpenFlow, but L3 and custom apps are our main focus.

That said, often times the subset of things that can be done w/ OpenFlow in actual existing hardware are things we can do natively. Feel free to email me if you want to discuss!

nolan@cumulusnetworks.com

http://www.pcengines.ch/apu.htm

By "open hardware," do you mean open source hardware, or something else?

From the responses, it looks like "open hardware" means something like "can install any OS on it" in the network switch world.

This is rather cool, and I'm glad to see some work being done in the networking gear space utilizing open designs and firmware like this. Unfortunately, unless I find myself needing to network a datacenter in the near future, it's not immediately useful to me.

That said, I do hope other developments in network gear that will be useful in other markets emerge from this effort.

From my perspective, there's a gap in network gear between the unmanaged, low port-count switches in plastic enclosures, targeted to home and small office consumers, and the lower tiers of Cisco's catalog, targeted toward top-of-rack or wiring-closet-of-a-larger-building type uses. I would love to see a managed switch with say, 8-24 ports, supporting features such 802.11Q VLANs. I would love to be able segment my network at home so different devices with different performance and security needs aren't all stepping on each other's toes. And I'd like the firmware and hardware designs to be open source, so it can readily patched when bugs are found, and easily adapted to new use cases.

I realize that I'm an outlier and that my needs are not common, or there'd probably be equipment on the market that met them. But it is my hope that as a result of Facebook's work here, and similar efforts, that building such a device will become feasible.

I also wanted something in the category you're describing and after shopping around a bit I stumbled across Mikrotik's products. I bought one of their 24 port smart switches and it was exactly what I was looking for. Not a lot more expensive than some of the nicer consumer-targeted gear, but it has way more functionality.

http://routerboard.com/CRS125-24G-1S-IN

Thanks for the link! Is it fanless too ?

the 15W consumption seems to hint it could be, but the datasheet does not mention it.

There are no fans.

That's the exact one I have. It's definitely not a "plug and play" router, but I love it. My only real beef is that most everything on it requires a much larger level of network administration knowledge than any other hardware.

I'm also scared that I'm going to open up my home intranet to the world every time I tweak some of the advanced settings.

Not trying to dissuade anyone from using them, but it's light years away from a DD-WRT-based router.

OTOH, the ability to run VMs on your router is magical. I have yet to do that, but I'm itching to do something like putting the unifi management tools on it.

Yeah, it definitely isn't something I'd recommend for people who don't know what they are doing. It's feature set is more comparable to enterprise hardware like Cisco or Juniper. Their terminal configuration utilities are pretty easy to use, but I actually really like their webui. It exposes all the functionality (which is a daunting amount) but is clean and very responsive.

That said, RouterOS pretty well documented and isn't hard as long as you know the basics. Mikrotik actually has a pretty vibrant little community around their wiki.

Can the firewall on that do outbound filtering?

I've been looking at trying to control some of my devices a bit more (eg, SmartTVs).

PFSense looks good, but the dedicated hardware boxes are kinda pricy[1], and I don't really want to build something myself.

A switch with a built in firewall at that price is an interesting option.

[1] http://store.pfsense.org/hardware/

Yeah it can. The firewall is standard iptables, so you can do whatever you want. It's also got DNS, http cache, SOCKS proxy and approximately 1 trillion other things that you may find handy as an all-in-one home/small business network device. Needless to say, it's maybe not the best idea security-wise to run all that stuff on one device, but I'm not terribly concerned.

I've heard people say it's not really powerful enough CPU wise to cope with a ton of rules (>hundreds), but I have a fairly involved firewall config on mine and it's no trouble.

Funny story, we use a firewall rule to punish roommates for not doing their share of the chores. If they get too far overdue on chores, we have an iptables rule to randomly drop a certain percentage of packets to their machine.

Looks like a very nice product (along with most of the other products). It's a little disappointing that they seem to do only the very minimum wrt the GPL:

http://www.mikrotik.com/downloadterms.html

I mean, sure, asking for 45 USD for a CD with the source is technically complying with the GPL -- but it does seem a bit strange in this day and age. Not to mention that for the source code to be useful, one would hope one could build a working routerOS image from it -- and it doesn't appear that the CD will enable a user to build a running image -- and therefore not facilitate changing the product.

But apart from that, my first thought was -- can this thing run BSD -- because pf is quite a bit friendlier than iptables (even if the latter has gotten a lot better lately).

Does anyone know if there are any recommended alternatives to soekris for running a bsd switch/router (preferably running at ~gigabit speeds) ?

Lots of PFSense people seem to think the PC Engines APU[1] is ok. Netgate sells a DIY kit[2].

It's only a 3 port system though.

[1] http://www.pcengines.ch/apu.htm

[2] http://store.netgate.com/kit-APU1C4.aspx

At home, I replaced my Cisco ASA 5505 (which replaced a Cisco 1811) with a RouterMaxx 1106 [0] running OpenBSD (from CompactFlash), though it's certainly not cheap.

[0]: http://www.maxxwave.com/solutions/security/routermaxx-6-port...

Not sure if it can run BSD proper. It should be able to in theory, there's nothing preventing you from flashing your own OS image, but I can't find that anybody has actually done it. There is a facility for virtualization as well. Not sure how desirable it'd be to run on bare metal though, as it uses dedicated hardware for routing that I think is proprietary, so it wouldn't work. If I was going to try it, I'd probably pick up one of their cheaper products first to test it out on (most of the lower end stuff is similar internally).

I'm with you about the GPL though. It's my biggest complaint about it. Apart from the proprietary hardware (which OK, I guess I can forgive it), they definitely seem to be playing it pretty loose with the terms of the GPL.

Pf is indeed nice and it'd be cool to run OpenBSD on it. That said, as I mentioned I really like their configuration tools and they make iptables actually quite easy to configure.

Your best bet for a BSD router is probably to pick up a cheapish computer and put in one of the Intel or HP quad-port ethernet cards and then plug into a dumb switch. They can be had for surprisingly reasonable prices, I saw some on Amazon for ~$80 (the quad-port cards, that is).

I've the exact same needs, and by occasion got hold of cisco SG300-10 switch, which I think fits the description - so I thought to share my experience, in case it is useful.

I use it for the home lab, where I have a one-armed router serving multiple VLANs, and I have 3x MacMinis running Linux as a "server farm" (The latter I use because they are quite a good gear power-management wise, scaling from ~18W at idle up to ~250W when all cores are busy, and because they are very very quiet, which is handy when the "lab" is next to the bedroom).

The biggest complain I have about this box is that the only way to manage is the Web UI, and especially the 802.1q configuration is a bit unintuitive (though I just learned the firmware is actually upgradable to something with decent IOS CLI, so I will try it out and update here the impressions).

Otherwise, needing just a very simple L2 switching and 802.1q trunking at gigabit speeds, and fanless operation - I am pretty happy with it.

8 ports works well in my setup (the main segmentation/trunking is really in the lab, the rest is either wireless, or directly connected to the "border router").

Where I needed to add more access ports, I used http://www.conrad.com/ce/en/product/976050/CE-Port-Network-S... to help. Also based on my experience seems to be quite a solid building block for a small home/lab network.

full disclosure: I do work for cisco.

EDIT: the upgrade to the latest firmware indeed unearthed the checkboxes to enable telnet/ssh, as well as quite a few new features, comparable if not more than the "bigger brothers". What's pleasant is a quite comprehensive IPv6 support.

Is the IOS CLI firmware official? If it is, do you need a support contract to upgrade?

It's a normal checkbox in the gui "Enable SSH" / "Enable telnet", so I would suppose it is.

But further tinkering revealed it seems to be a quite-close approximation of IOS, but not the same IOS you'd get on the "older brethen" boxes.

It's about 95% the same, with differences in small details - the format of the output, the behavior on "more" prompt, the look of that prompt, the way the certs are stored, etc.

Nonetheless should be close enough to be usable in a geek home environment.

I got the software here, looks like it allows to download w/o logging in: https://software.cisco.com/download/release.html?mdfid=28301...

Since the box I had ran a rather old version, I had to do the intermediate steps as described in http://serverfault.com/questions/622724/error-illegal-softwa...

One caveat I noticed is the ssh seems to not work when connecting from Ubuntu 14.04, works fine from OS X. When I have time, I'll debug it further. (I very rarely do any changes on it, the CLI was more an unexpected bonus I wanted to check out rather than a real need).

The specs on this look very nice indeed. I'll be taking a closer look at this.

My main hesitation is the proprietary firmware. Now I'm not going to disagree with anyone arguing that Cisco knows what they're doing and is competent at putting together firmware for the hardware they sell. Nor will I disagree with anyone arguing that open source is not a magical talisman ensuring quality. No, my main concerns are a) timeliness of critical updates, and useful lifetime of the hardware vs support lifetime.

Being open source cannot prevent bugs, but once found, fixes tend to become available quickly. Also, I have found that the useful lifetime of computing and networking hardware tends to exceed the period of time the vendor will offer support for it. I have gigabit ethernet switches I bought years ago that still work just fine, even though they're no longer sold. That's what I love about OpenWRT. The hardware my home router uses is discontinued, but still does the job just fine, and I can still get updates when I need to.

That said, I think I will be checking the SG300 out. Thanks for the recommendation.

I've sent the software download link in the other comment - so far the history shows quite regular software updates for this box. But I share your concern and can not say much about this box besides what I can infer from the software publish history.

+1 on the OpenWRT. Building a custom package that allows you to get a $20 specialised networked appliance is a breeze. That platform absolutely rocks.

The 2960 series has 8 and 24 port models, that are fully managed switchs - VLANs, management, anything that you can do in the IP Base IOS.

See the bottom of this page for models the 24 and 8 switches are what you are looking for. http://www.cisco.com/c/en/us/products/switches/catalyst-2960...

Also, if you want a cheaper options the Dell Basic switches are fully managed and have just about any feature you could want.

http://www.dell.com/us/business/p/powerconnect-2800/fs

Unless I'm missing something that you are looking for.

Got a used 24 port 2960 on ebay 10/100 ports for ~$65. Maybe doesn't mean the "open" portion, but the rest is definitely satisfied. Also, bought it 5~ years ago and still runs beautifully (Noisy fan though).

There are some good cheap managed switches out there. I have a Linksys SFE2000-something 24 port PoE switch for my PoE devices (security cams, IP phone, other stuff) and a TP-Link [something] 16 port switch. Both support basic VLANs, SNMP, etc.

I have 1 2port VLAN that connects my cable modem to my router on the TP-Link. From that switch I have 1 port cables to a machine with wireshark, and I can configure that port to monitor various VLANs for whatever reason.

I run MRTG for several things, including basic traffic graphing. I display the graphs from the router uplink port and a couple of other key ports in a window on the VMS that also has security cameras on it. From that monitor I can keep on eye on key things (cameras, Internet I/O, some home automation stuff).

Anyway, I haven't found an affordable "perfect" switch for home stuff, but there are a lot of cheap, decent managed switches that give you a lot more flexibility beyond "everything on 1 network".

My home net is essentially segmented into Primary LAN, Security Devices, Guest LAN (mostly just a wifi bridge) and LAB LAN.

Have you found an OpenWRT firmware equivalent for these devices? A lack of an open-source firmware for such a switch has been one of my hangups. I'm leery of the firmware that would come with such a device, especially given the history of the firmware of home routers being implemented poorly (bufferbloat, obsolete versions of software with known vulnerabilities, a lack of upgrades available from the vendor because they've EOL-ed support on the device to make room for the new shiny version -- which is also shipping with vulnerabilities).

I think what would be ideal is something in the vein of the Linksys SFE2000, with an open-source firmware akin to OpenWRT. AFAIK, that, or something approaching that isn't available. However, I would love to be wrong in that regard.

This isn't really suitable for the purposes you're describing, but it has a niche for firewalls and routers that may interest someone. I've been using PC-Engines Alix[1] boards for this purpose since 2008. They are 1-3 ports depending on model, but with VLAN tagging a few ports can do a lot of routing or other traffic processing. I run OpenBSD on mine, but they should work well with anything that can run headless.

[1]http://www.pcengines.ch/alix.htm

Checkout the ubiquiti switch:

http://www.ubnt.com/accessories/toughswitch/

and

http://www.ubnt.com/unifi-switching-routing/unifi-switch/

That's exactly what you are looking for. SOHO friendly (fanless, cheap, supports vlans).

Wow. That switching fabric is so much smaller than the HIPPI crossbar backplanes we used in the 100baseT/pre-GigE switches that I used to write firmware for... almost 20 years ago now. Of course, complexity means higher cost, so it's obvious why nobody bought our stuff. Hopefully, this open hardware project will have better luck!

/* it's easy to forget just how many iterations of Moore's Law have happened since the mid-90s */

The work going into whitebox switching and routing is super exciting. I really dig this stuff.

I get the feeling this is a bit like Bugatti building the Veyron, its super cool and impressive but the vast majority of us will never be able to or have a need to drive it.

It is still fun watch this switch go around the proverbial track, but I'm happy knowing that I'll never have to configure, build and test a switch of this complexity unless I really absolutely have to, with my largest caveat being AWS disappearing from the face of the earth.

You're not the target market. You shouldn't be making an analogy with a Veyron, which remains a luxury car for an individual. You should rather be comparing it with a firetruck, a vehicle custom-built for a purpose you as an individual you will never need.

I currently work in the network hardware industry (think Cisco, Juniper, etc). Our boxes sell in the 6-figure price range, each [1]. We sell to your ISP, wireless carrier, datacenter constructor. We're the competition this kind of box is aiming at.

[1] it's highly specific hardware and software for a low-volume market. Individual chips used in the hardware can cost multiple thousands of dollars each.

Good points. I doubt if they are aiming at any competition. Their stated goal is to reduce the cost of the hardware for themselves by standardizing it. Their hope is that the other big guys (amazon, google etc) will pick this solution up driving up adoption and lowering the cost. If this happens, Cisco, Juniper et al gets disrupted and that's just a side effect.

> Individual chips used in the hardware can cost multiple thousands of dollars each.

and 10 years ago, the compute power of a QC i7 would have tens of thousands of dollars.

Right, but we don't care about past chips. Moore's law applies to the whole market at the same time, so today's special network chips are still much better at what they do than today's general-purpose chips.

It's kind of stupid to say that today's general-purpose hardware can do what yesterday's special hardware could do at a faction of the cost, because the specialist market has also moved on and wants the performance of today's special hardware.

MRV?

One of the great things about computing is that stuff which was insanely high-end N years ago is high-end of affordable now and will be commodity in another N years.

And N years after that you get that functionality for free along with your new TV.

If the people who brought me HDCP[1] are the same people that will give me a networking switch inside my television, maybe I will consider using this Facebook switch after all.

[1] - http://en.wikipedia.org/wiki/High-bandwidth_Digital_Content_...

I'm not entirely sure what I'd do with 1.28 Tb/s of switching power.

Though, 30 years ago, in Neuromancer, William Gibson wrote about the city of Los Angeles transferring megabytes (yes, whole megabytes) of data each second, so he probably couldn't have imagined what someone would do with the gigabit switch on my desk.

Does anyone know which team in FB is building software for these products? I did glance FB company info but could not get the information.

I no longer work there but I can ask the usual suspects. Email is in my profile.

Thanks and sent you an email. For anybody who is looking for similar details on the team who are behind these products, just FYI - there is not much info on FB site or on the internet. If I get any info later, will report back.

The only datapoint I have is that it's in the SF bay area, since I am acquainted with someone who got poached from Amazon specifically to work on this and had to relocate from Seattle to there.

I just want a fully open source switch that's ready for enterprise, from the h/w to s/w. Linux preferred but bsd based is acceptable.

I've been a "cisco guy" since 2001 or so, but I am so tired of them. Licensing fees kill budgets that could be used on other things, and you end up surrounded by consultants that only ever touch Lozedoze systems insisting that "nobody ever got fired for buying Cisco". Smartnet is a must for some equipment, yes, but I'm so ready for a paradigm shift in networking.

I've really been watching ubiquiti and their switching/routing products, they seem very promising but not quite prod ready. I am impressed with Dells open switches too.

Edit: The Microtik stuff being linked elsewhere here is looking pretty awesome too. Not quite FOSS but still.

So this is very interesting.

As someone who runs a 6509 (almost fully populated with line cards and specialized controllers) as my HOME core switch (with a second one on the way for full redundancy), an open modular switch is very cool.

(I'm currently hacking on OpenFlow -> NETCONF bridging, to bring typical SDN capabilities to legacy Cisco environments). Hence my rather..... extensive home network.

This is very similar to what Cisco is doing with Nexus and the "fabric extender" TOR replacement kit. I'll have to see if Facebook has any of this stuff in GIT and stand it up in a VM environment and play with it (I already do a bunch of OpenFlow stuff on OpenWRT and am looking at implementing an open southbound API on FPGA on the parallela board).

I was expecting a link to something like this, but more advanced

http://opencores.org/project,mac_layer_switch

and some links to digilent dev boards for hardware, however the linked project is a bit more ambitious, all full of custom ASICs and such.

An open source switch made out of COTS FPGA dev boards would be interesting. So you'd use something like

http://www.digilentinc.com/Products/Detail.cfm?NavPath=2,400...

But what to use as a COTS "just unpack it all, plug it all together, upload the firmware and go" backplane?

That NIC seems quite overpriced for 4x1G...

It's not a NIC; it's an FPGA that can be programmed to do things that NICs have never dreamed of.

I understand that, but you can get more powerful ones for not much more money.

NICs or FPGA boards with NICs? If the latter, I'm all ears

Netronome, http://www.mellanox.com/related-docs/prod_adapter_cards/PB_P..., there are others for more $$ using the Altera Arria 10, or other Xilinx FPGAs with 2x40G or even 4x100G.

Good thing it is not targeted at people looking to buy a 4x1G NIC, but at people who want a nicely integrated FPGA dev plattform with 4x1G and fast connection to a PC.

Isn't this just wrapping for the COTS switching ASICs or am I missing something big?

Almost all "open" projects seem to hit a point where the line between open and closed is quite arbitrary, even if they choose not to see it that way.

"Wrapping commodity ASICs" is worth $4B: http://www.arista.com/

If Facebook wants to give me that value for free I won't complain.

And the founder/chairman/chief development officer of Arista, Andy Bechtolsheim, is a board member and founder of the Open Compute Project.

FWIW, all Arista hardware is AWESOME. They run a lightly modified very minimal linux operating systems that is very clearly (not even really hidden) Fedora. I got the privilege / opportunity to reverse engineer one of these to see if it was hackable (ie: add custom hardware into them or run apps directly on the switches).

Pretty much the entirety of the Arista userspace is python with all of the magic really in their ASICS. Truly, I've not seen many better engineered pieces of equipment. At the time (I did this > 4 years ago), the Arista switches had dual core AMD Athlons with a few G of ram. They were willing to put more RAM in one, or let us upgrade it. For a so called "vendor product" Arista clearly gets it. They have some really solid Linux guys working for them. When you get the pleasure of using their python apis or pull one apart, it shows that they know what they're doing. It isn't completely insane to think of running apps directly on the switches when they are simply x86_64 Fedora boxes with some fancy asics and a lot of interfaces.

pretty sure the big win here is:

1) low individual component cost (which is great when you need 100's of them for a datacenter build out) 2) no frills, high throughput non-blocking backplane which are normally only available from the enterprise grade network vendors at top dollar per individual component, and come with a mountain of features and bugs you DONT NEED 3) software defined networking stack: if you can imagine a scenario/feature that would improve your life, nothing but development costs will get in your way. contrast with enterprise vendors, which requires explaining the problem to begin with, and dangling a wad of money explaining how it will be worth their while to develop it

This looks great. They also have the advantage that James Hamilton mentioned AMZ got from building their own hardware - the get to do a cleansheet s/w design and not have to support all the gazillion options that Cisco does.

The talk was on HN a little while ago, worth watching: https://www.youtube.com/watch?v=JIQETrFC_SQ

The scale at FB means they're probably saving hundreds of millions doing this.

Am I the only one who read the title and thought it referred to a 'pushbutton' style electronics hardware switch? I would have been quite excited to see that!

I wonder if youtube will ever notice that an uploaded video is quiet and only on one channel, and allow for normalized mono listening..

Reply on news.ycombinator.com