New Optical Form Factors for 400 Gigabit Ethernet | Hacker News Reader

New Optical Form Factors for 400 Gigabit Ethernet

lightwaveonline.com

121 pointswalrus0177 comments

The real debate is between OSPF and QSFP-DD as they are the only ones with a shot at high density.

QSFP-DD is backwards compatible so that you can run your 400G port as a 100G or 40G with 4 lanes at 25G/10G. OSFP will be able to do the same thing but with an adapter from OSFP to QSFP

But what this article fails to mention is that OSFP is the connector of the future that will support 8 lanes of 100G to support 800G ethernet with good signal integrity and capability to support high power optics for longer distances. ( This looks to be in 2020 )

So QSFP-DD is a 1 generation connector...

OSFP will see us through 400G and 800G.

Disclaimer I work at Arista Networks who is bullish on OSFP.

For more info see Andy Bechtolsheim's talk on 400G optics at OCP

https://www.youtube.com/watch?v=Kotu6B7AQpk

I am also optimistic about OSFP as compared to the QSFP size for 400G. I'm not a laser engineer but I do have a pretty good understanding of how 400G will be achieved through two strands of singlemode. For medium reach optics it's actually an 8-channel CWDM with prism built into each optic, so the optic body needs to be large enough to dissipate the heat from eight separate 25 Gbps lasers. CFP size will obviously handle that but is a true first generation solution and is huge.

The only bad thing about OSFP is that its highly similar name will confuse the hell out of network engineers who never see the OSI layer 1 of a network, and deal with things at OSI layer 3.

OSFP != OSPF (the routing protocol).

100G/wave device will be the norm moving forward and 8 lane devices for 400G won’t be around much except in niche applications.

for long reach, yes, but not for ISP-to-ISP interconnection... the cost of coherent (QPSK, 8PSK, 16QAM) 100G SerDes and modulation/demodulation is considerably higher and only justified if going a great many km. For intra-peering-facility connections I don't see coherent 100G (or multiples of coherent 100G bundled by CWDM into 200G, 400G interfaces) becoming the norm anytime soon.

I was looking at this the other day; it sounds like 100G-DR1 and 400G-DR4 are not coherent and vendors are aiming to make it commodity. Coherent is more like 200-600G per lambda.

At present 100G-LR4 is the defacto standard for ISP peering interconnections (such as to an IX switch in the same building, or for PNIs between big players within the same carrier hotel). Because it is very low cost. The cheap 25 Gbps x 4 approach and low cost cwdm prism are a big part of that. Since 100G will soon move from qsfp format down to the same size as SFP and SFP+ (fitting 48 in a 1RU height, 17.5" wide line card in a chassis based Arista), I don't foresee them becoming any less popular... Not when the optics are so affordable.

We aren’t talking about long distance and coherent here but direct detect. Coherent is not cost or power competitive with direct detect for short reaches yet.

When I talk about SERDES I’m talking about the electrical signal interface to the optic. Also it’s worth noting that the majority of coherent DWDM interfaces are some kind of embedded optic already.

Sorry but OSFP has at most a life of 2 generations and likely only useful for a single generation. 112G SERDES is not going to scale well and will require repeaters all over the place. Moreover doubling the signaling rate to 224 Gb/s is wholly impractical. The future is embedded devices and not legacy pluggables.

I get the attraction of OSFP, but in early days backward compatibility usually trumps future potential. If I'm buying an high density switch/router tomorrow (figuratively speaking), I'd greatly prefer the flexibility to use the density it provides for some existing lower speed connections. As long as QSFP-DD 'works', as a customer I'm not really concerned with which technology is more elegant or technically 'better'.

The problem this presents for OSFP going forward is that it encourages early adoption to steer toward QSFP-DD, which will tend to drive follow-on purchase decisions as well.

I see any criticism of OSFP is getting downvoted but your points are correct. OSFP is not needed for 400GE.

There is no real debate between QSFP-DD and OSFP. OSFP will exist at Google but nearly everyone else is going to use QSFP-DD. The main reason is backwards compatibility for data center upgrades. The only reason to use OSFP might be because you want to use a 400G-ZR plug. But with the advent of 7nm DSP, vendors will be able to fit the ZR into the DD plug.

So if the above is true OSFP is not needed now.

Honestly, 800G is only incremental at best and frankly it’s not clear that An 800G pluggable is the way to go. The 800G presumes 32 radix and that breaks down power-wise in the datacenter when we get to spine switches where we need 128 radix. The reason being that we need so many power hungry chips to make a high radix switch fully non-blocking (eg Clos topology). 112G SERDES will have limited reach and hence require many retimers which eat power too. So the OSFP is in my view is A solution looking for a problem.

I remember having to schedule a 780Kb download overnight and hoping no one picked the phone up. Or years later having to download shows in advance (illegally because they weren’t available) because streaming wasn’t possible.

Now I casually (and legally) boot up any show I feel like within seconds at HD resolutions. I love thinking about the sheer amount of data that is traversing around the internet now. And it’s more fun to imagine what becomes possible when you bump up another order of magnitude or two.

In games for instance, a significant limiting factor for implementing large player counts in fast-paced multiplayer come down to bandwidth. The amount needed, for the most part, scales linearly per-player as the player count increases. Add more bandwidth (and some more CPU power) and games can evolve entirely new experiences.

The battle royale games like PUBG are a great example, but due to bandwidth issues and CPU usage the server tick rate is much slower than normal fast-paced games (20hz vs 60hz) and therefore the experience suffers quite a bit.

Multiplayer gaming has more issues with latency than a lack of raw bandwidth.

Not necessarily, depending on how the game is implemented a player with bad ping can be the only one who suffers the majority of the effects of their bad connection.

I am currently working with this tech:

https://www.photonengine.com/en-US/Quantum

And it does deliver on their promises, impressive stuff.

This looks neat, think it'd do well with upcoming projects like Star Citizen from CIG? Physics sims have to be limited dramatically atm.

Latency is usually from bandwidth being queued somewhere along the path

60hz even is low today by competitive standards for FPS games. 128 is nice, it should be a minimum.

Often it’s only on LAN games for tourneys where the tick rate is that high. With internet latency or bandwidth requirements at that rate you are going to start dropping more and more packets and suffer from potentially more client side mispredictions.

Packet loss is usually very low. 1% and the game will most likely be unplayable.

Sending twice as many packets and losing 1% of them should be playable.

Maybe if you send some packets twice. Otherwise no. Packet loss is deadly.

I would go so far as to say that netcode for an FPS is objectively wrong if it's not sending data in a redundant form that can tolerate lost packets.

Tolerate yes, but it will still have an impact. In case of an online shooter, resending it will cost an additional RTT. If that resend is even happening, the next packet will make the dropped one obsolete.

Resends in an FPS are for dire scenarios, not 1% loss. 1% packet loss can be easily obsoleted and compensated, and the general performance of having 126/128 packets can be better than 60/60.

It's not frequent that I experience much loss when playing 128t servers. Mostly zero, very occasionally 1%. I wouldn't be paying monthly for it if I knew it to happen frequently.

My biggest gripe with ESEA is that I cannot have Hyper-V enabled on the OS that I am playing from.

I don't think there's a regular FPS game that does 60Hz. That's usually reserved for the private servers, tournaments.

This guy does a lot of game network analysis of the most popular FPS games:

https://www.youtube.com/user/xFPxAUTh0r1ty/videos

I think you mean 128Hz as most games have settled on around 60Hz for any sort of competitive FPS game play. As per the person you linked, Rainbow 6 Siege, Battlefield 1(and 4/Hardline, presumably), CS:GO, Overwatch, CoD: WW2, and Quake Champions are all at least 60Hz for the client and server, while Quake Champions is 144Hz for the client. The only game I can think that uses 128Hz for both the client and server is dedicated servers or private matchmaking clients for CS:GO, and the game using less than 60Hz for the server are usually smaller games or battle royale games like PUBG and Fortnite.

Meanwhile the batte Royale games are throwing that all out the window. Epics engine is struggling to maintain 20Hz on both PUBG and Fortnite though seems they are making advancements.

CS:GO ESEA or FACEIT are always 128t. While in CS you don't see a lot of the issues at 60t, you do find them occasionally in high level play. I wonder if the same is true for R6: Siege, in which I feel interp is the cause of a lot of my deaths. The desync in Siege is absolutely insane, and I've lost many fights that were round or game changing to it at diamond. Even PUBG feels more responsive at 20t (the same tickrate as Minecraft, I like to note.) other than the occasional fight that ends when you kill someone as a bullet hits you, registering the blood splatter for you while you take no damage.

Most games are 60+t these days, the outliers being PUBG and Fortnite. Competitive CS:GO is the major one pushing 128t, and it's very noticeable at high levels of play. I have lost many firefights due to the weird interp that R6: Siege has, that simply don't exist at 128 CS fights.

TF2 has this issue where people can change their interp in a certain fashion that allowed hitscan guns to have a broader headshot range. I'm not sure if it still has it, I haven't played TF2 competitively since 2012. Looking at R6, nothing ever seems to line up properly between two clients for pixel-peeking, fast swings, or slow peeks alike. Something feels off about it compared to other shooters, but I can't pinpoint it. It feels like it's interp as it mimics the issues other games had related to interp.

For reference, I don't want to seem like some random person who just does badly and blames it on networking mechanics, I'm diamond in siege, A+ in ESEA and global in mm, held top 10 on the leaderboard in PUBG. (It also does feel nice to be able to say that sometimes, even on a technical site like this.)

I've seen his videos a lot. Also Escape from Tarkov, if you see in his videos he has listed at 93hz. In reality, the game has horrible issues that cause clients to experience desync, allowing delays nearing 1 SECOND in reality.

Tarkov used to have an issue where if you repeatedly opened and closed specific doors on some maps, it would create desync that was ever-growing. You could sometimes end up dying 5 minutes in the 'future,' while your client is 5 minutes out of date. It doesn't happen anymore thankfully. The game is plagued with other issues though.

Bandwidth is not the issue in PUBG.

Yeah exactly... while higher bandwidth may open the door for some new things in games... I think it's possible there is not a single game out there that really needs even as much bandwidth as a single HD video stream. The limiting factor is really latency and jitter.

I wonder when we will start seeing > 1Gpbs Ethernet in consumer products. I've read about initiatives for 5 or 10Gb consumer Ethernet. I'd love to be able to push/pull from my NAS at > 100MB/s.

In-wall cabling ends up being the limiting factor for a lot of this -- definitely check out the Nbase-T stuff coming out, which can negotiate from 1Gb up to 10Gb depending on what the cable can support (Ubiquiti just announced an AP using it, Juniper should have switches coming out this quarter IIRC).

10Gb ethernet works fine, but the power requirements for copper (base-t) are high and SFP+ is a bad standard for consumers (finicky connectors, low mating cycle rating, etc).

It's a fair question; I suspect most of the reason for the lag is that consumer internet connections rarely offer >1gbps, consumer LAN applications requiring >1gbps of traffic haven't emerged (for NAS/SAN, a directly connected USB 3 is probably the way to go for 10gbps transfer, which is what you're looking for here), and WiFi - how most people are connecting their consumer devices to the network - doesn't offer more than a gigabit of goodput in almost any consumer configuration. (e.g. 802.11ac with two spatial streams and 80MHz channels gets ~700mbps at MCS8)

That said, the NBASE-T and MGBASE-T efforts did consolidate to produce a 802.3bz standard that can carry up to 2.5gbps on a Cat 5e line and 5gbps on Cat 6, and we are starting to see consumer and SMB switches, APs, and NICs that support bz.

Moral of the story is wire for Cat 6 and look to upgrade to bz in the next year or so.

one of the major use cases for 802.3bz is high bandwidth WAPs, specifically things that can actually pull more than 1 Gbps of data... to actually achieve that with wifi standards usually requires:

1) dual, separate 2.4 and 5.x GHz radios

2) 3x3 MIMO

3) 802.11ac wave2 (1024QAM)

4) use of really wide channel sizes, even an 80 MHz wide 802.11ac channel won't get near 1 Gbps aggregate throughput, needs to be the ridiculous 160 MHz channel size in the 5.x GHz part 15 frequencies.

5) all of the above combined, or more than two radios in an AP such as expensive high density Xirrus or Ruckus WAPs which can have two or three separate 5.x GHz radios in one physical body.

Spot on...and it is going to be a while before a typical home has those needs or capabilities. Broader FTTH could change bz and ac wave 2 adoption curves as people end up annoyed that they are paying for a gigabit but not actually receiving it.

I see how cat6 works for economics of scale; but when I last year looked to price 10gbs switches for a lan for a school (with a meagre budget) - I was surprised at how expensive second hand 10gps gear was. In the end we stayed with 1gbps even though we had some use-cases where 10gbps would make sense; mostly budget San/nas for video edit/backup. And to a lesser degree audio editing.

I think the 1gbps Cisco switches were hand-me-downs from a university - but I saw no indication of reasonably priced 10gbps switches with minimal management (we didn't need "all" of Cisco ios - but minimal remote management and steady performance would be nice).

We also threw some lan parties, so budget soho stuff that'd fall over easily wasn't really an option.

Cost. 1Gbps Ethernet Router, Switches, SerDes, Controller, everything is a magnitude cheaper then 10Base-T or NBase-T ( 2.5/5Gbps Ethernet)

Apart from some Prosumer and Business usage there just isn't enough market to scale this down to a cost competitive product. I think it may possibly remain as a niche. ( Please Prove me wrong as I too want my NAS to transfer at 500MB/s )

I saw some article suggest Small Cells in 4.9G/5G or 802.11ax WiFi will be key to this acceleration, both are coming in 2019.

SMB3 multichannel does allow you to exceed 1Gbps across multiple bonded links. If your NAS has 3 or 4 1Gig ports, that might get you there, with an intel 10Gig card in your desktop and a 10Gig-capable switch.

> I'd love to be able to push/pull from my NAS at > 100MB/s.

At a guess you are not a typical consumer. 10G will happen for the consumer, but 100G might never happen for the consumer. And I suspect it will be a while (quite a few years) before 10G will become commonplace enough that economies of scale kick in making it affordable.

Which kind of sucks now that single SSD's are commonly capable of speeds at multiple Gb/s.

One of the big problems with 10G is heat. Common 10G copper cards use 10+ watts and have considerable heat dissipation.

> Which kind of sucks now that single SSD's are commonly capable of speeds at multiple Gb/s.

Single SSDs are often capable of multiple (low) tens of Gb/s. Like Samsung 960 Evo, about 25 Gbps reads (== 3.2 GB/s).

What are distance limitations at 400?

Trans-Pacific :) But more seriously, this article is about the device-side electrical interfaces -- the modules will transceive on any one of a number of physical interfaces depending on the specific needs (ultra-short-range for in-rack/in-DC, though that's a less likely application of this technology). Most multi-lane paths like these are using multi-strand optical cabling for short-range, and typically multiplexing the signal using on-board DWDM in/around the 1550nm band for two-strand long-range (which is primarily constrained by the power available to the module). This band is used because it can also be amplified inline (ref: erbium-doped amplifier) to extend the range.

(this is an oversimplification, I am not your network engineer, I am not a network engineer, consult a qualified network engineer in your jurisdiction)

Does aanyone here work with consuming (not just passing it kn) these amounts of traffic? How do you manage to get anything meaningful out of it?

network engineer here: nobody connects a 200 or 400GbE interface linecard off a router to a single anything... The purpose is for ISP-to-ISP interconnections for truly huge amounts of traffic. For example direct peering in the same IX/carrier hotel between a huge source of content (youtube/google, netflix, etc) and a huge ISP (comcast, centurylink).

For a big ISP, also things like 100GbE connections from a city's pair of core routers to slightly smaller aggregation routers.

edit: I don't know that anything "consumes" data in the way that the questioner is asking. It's more about the aggregate amount of data. One netflix 4K stream to an xbox one s is about 15.75 Mbps. Now multiply that by a hundred thousand netflix subscribers in a typical comcast service area, any of whom at any given time might be sitting around and watching Altered Carbon or Breaking Bad.

The only potential consumers I can imagine are like the Great Firewall of China traffic monitoring systems.

Or the NSA: https://en.wikipedia.org/wiki/Room_641A

Raw high frame rate video can into that range.

Phantom v2640 for example does 6600 frames per second at a at 2048 x 1952 resolution = 26 Gpx/sec or something like 100Gbit.

Currently they simply buffer everything on Camera and use 10GigE, but in theory they could look into 100+GbE connections for newer designs.

upcoming on-camera flash memory interface

https://en.wikipedia.org/wiki/CFexpress

basically a way of putting a M.2 interface NVME PCI-Express SSD into a camera.

https://petapixel.com/2018/04/09/prograde-digital-unveils-wo...

According to the link that's currently limited to 2 GB/s which far to slow to use for this. Phantom v2640 needs ~50+x that speed to do it in real time and I assume the bandwidth needs will keep increasing on the next model.

So what storage device does it use currently? except for RAM, there is not much faster than a NVME based, PCI-Express 3.0 x8 interface SSD.

If it's some truly ridiculous bitrate it probably uses proprietary packs of RAM with battery backing.

Ok, found more info on that model right now it's got up to 288 GB of internal RAM. At 12 bit per pixel which is lower than I thought that lasts 7.8 seconds for ~40Gbit/second.

Anyway, as this is all internal so they have a lot of options.

yeah, if we're talking about RAM, there is still a significant difference in MB per second write ability for something like DDR4-3200 vs even the best NVME pci-express flash-based SSDs.

Cumulus (disclaimer: I co-founded it) has customers that use these: http://www.mellanox.com/page/products_dyn?product_family=260...

to dual 100G attach their servers to a pair of ToR switches. As a result, they'd really love to have something faster than 100G for their ToR to spine uplinks, but that isn't quite available yet.

Insanely (and awesomely), Mellanox already has these: http://www.mellanox.com/page/products_dyn?product_family=266... dual 200G NIC.

100G NICs and dual port of the same version are a really good argument for the need for more pci-express 3.0 lanes in single socket servers. The AMD epyc is a step in the right direction. If you assume a server that might have two m.2 nvme SSD in it, those take up lanes, and then the nic will eat an x16 interface.

Network Engineer here as well - I support one of the world's largest enterprise organizations (although my area of 'expertise' is pretty far down the stack - somewhere between the internet and the WAN circuits) - I can see no personal use for a 400 GbE port on anything.

As far as my organization goes, there will always be a bottleneck somewhere else (speaking location to location across the WAN).

The only applications I could see this being useful would be maybe huge ISP interconnects or maybe some big players in the video streaming world. That doesn't mean that future-proofing is a bad thing - of course cost will probably be a very large concern in the case of 400GbE...I see a lot of 40 gig ports that go completely unused, personally.

It's not hard to imagine 100GbE being used inside datacenters. AFAIK, 40GbE top of rack switch is pretty common.

I've seen some 160GbE links in datacenters - 4x40's - how much they were actually utilized, I have no idea.

if you have a really beefy single bare metal hypervisor (typically xen or kvm) it's totally possible to give it 100GbE to the switch now:

http://www.silicom-usa.com/pr/server-adapters/networking-ada...

I don't have anything this serious in my own production use, but I'm thinking of machines like dual socket x 32-core per socket xeons, or amd epyc, with 512GB to 1TB of RAM.

then pass vlan tagged traffic off one switch port, to each specific guest VM, of which there might be dozens to 100+ VMs, as subinterfaces of that.

HPC Clusters would be one candidate

We use 100 Gb EDR Infiniband to each host in our HPC clusters and with a combination of file system access, message passing, and general network traffic we actually use that bandwidth.

Can’t you saturate 100GB easily with a single disk array these days?

Yes, I've worked with several that can do that. The interesting thing with disk arrays now is the controller heads are becoming the bottleneck.

You used to have many many shelves of SAS drives per controller pair, now the SSDs are so fast that the CPUs are saturated and you have to scale out, not up.

If I can't (reliably) handle 100GBps coming from the RAID array it sounds like I'm not going to be able to handle 400GBps coming in from the NIC.

So these really are just for interconnects for the time being, until server hardware gets faster.

There are some applications which can handle 100Gbps and potentially higher speeds on servers. Usually they require using DPDK or similar frameworks with userland TCP stack (and thus are usually pretty specialized), but at least Netflix has managed to serve 90+Gbps of TLS traffic using "default" FreeBSD TCP stack (https://medium.com/netflix-techblog/serving-100-gbps-from-an...).

Not working on anything like that myself, but if the direction going forward it to stream not just 4k TV, but also 4k video games (console in the cloud), you will need some healthy backbones (even if they still run only IPv4 in 5 years!)

Streaming uncompressed raw video over ethernet for TV production. It's still commonly done over dark fiber but people are slowly migrating towards IP. A single uncompressed 4K video stream is over 12Gbps, 1080p is 3Gbps and obviously when you're producing content you have a few of these running alongside each other (multiple cameras, feedback, big screens etc... Think something like the world cup or superbowl). It quickly adds up to a tremendous bandwidth. On top of that it needs very low latency and jitter to keep all the streams synchronized.

Why would anyone stream uncompressed raw video? Even MPEG-2 at q=1 with no B-frames is about 1/10 the size, visually lossless, and adds only a few ms of latency.

Looks like for editing you're looking at up towards 3.5 gbps per camera, not sure how much to add for sound. So a three camera production could easily push over 10 gbps. Pushing past 100 seems to be a bit of a stretch. Maybe for 3d?

5G base stations cite 50 Gbps bandwidth, connect a bunch of these together in an urban area and you might want 400G for their uplink/backbone/whatever it's called.

More realistically as cellular carriers are upgrading rooftop sites to the latest LTE gear, what I'm seeing as a carrier, they're fed by either a dedicated 1G fiber transport back to an aggregation site, the same at 10G on fiber, or point to point 60GHz (1-5 Gbps) mmwave, or 71-86GHz band ptp mmwave (1 to 10Gbps).

It sure sounds like 5G deployment is "realistic". Big operators have announced deployments & limited availability this year.

Yes, in my company we generate and receive uncompressed video in software and get to these kinds of bandwidths from a couple of servers running dual 100GbE: https://www.youtube.com/watch?v=A4L5xEXXlas

I'm not consuming it, but I can produce over 100Gb/s of traffic from a single CDN node if it is equipped with multiple 100GbE NICS. So 200GbE and 400GbE are exciting to me.

(for context: https://medium.com/netflix-techblog/serving-100-gbps-from-an...)

disclaimer: I work at IBM Research

we have just finished setting up our new big data crunching system which sits on 48x100GbE links on the full flash storage side and on 32x100GbE on the client side (split 100GbE -> 4x25GbE).

at this point we are saturating most of the links.

Reply on news.ycombinator.com