Elastic Network Adapter – High Performance Network Interface for EC2 | Hacker News Reader

Elastic Network Adapter – High Performance Network Interface for EC2

aws.amazon.com

82 pointsjeffbarr10 comments

Does anyone know what type of hardware they are using to accomplish this? Is it just your standard Intel X520/540 10GbE adapter, or is there some secret hardware they have access to?

FWIW, saturating line rate 10GbE has been problematic at times in the past due to challenges like single-threaded drivers, which could only run on 1 core at a time, even if there are 128 logical cores in a system (like the X1 instances). There are also challenges offloading VXLAN encapsulation to hardware at 10GbE line rate, but it seems like AWS has figured that out.

It's from http://www.annapurnalabs.com/ so it's probably a custom(ized) ARM SoC. See http://thread.gmane.org/gmane.linux.network/416446 for more details.

Since it offers 20Gbps to the VM, i'd guess it's a 25GbE NIC, since those and accompanying switches have just come out.

VXLAN in the NIC (and the switch) at full line-rate offload has been a thing for at least 18-24 months.

Most Modern NICs also support all of the features listed here, like checksum offload, Receive steering, multi-queue, etc. Check out Mellanox CX4/5, Intel X710, QLogic, and Netronome if you want fancy stuff. They aren't even that expensive anymore.

Their Linux driver has support for every current and future standard: 1/2.5/5/10/25/40/50/100/200/400G, nice future proofing.

They also have a DPDK driver: http://dpdk.org/browse/dpdk/tree/drivers/net/ena/base/ena_co...

Thanks, but my guess is that it's actually a bonded 2x 10Gbps link, which would also give you 20gbps, and cost a lot less per port.

I'm seeing 25G at the same price as 10G, but I'm not using Amazonomics.

With current per port costs and economics, that doesn't make much sense. The $/Gbps/Watt improves with every spin of the chip, and every increase in serdes bandwidth.

10GbE (64x10G) - ~$4,500, $70/port

10GbE (32x40G) - ~$6,000, $50/port

25GbE (72x25G) - ~$7,000, $100/port (Seen as low as $5k)

25GbE (32x100G) - ~$9,000, $75/port (Seen as low as $7k)

2x 10GbE = $100

1x 25GbE = $75

Not to mention the extra cabling costs, and complexity.

Huh -- why is Amazon introducing ENA as a paravirtual virtual device? They already provide Xen netdevs to VMs.

The Xen PV device would be able to use the (more efficient) xen bus/event channels for interrupts vs. ENA (MSI-x, by looking at their driver on the Linux netdev mailing list).

The Xen PV device already also supports multiple queues for xmit, see netback_init() in netback.c.

2. The ENA LLQ interface (pushing short packets via CPU write-combiner writes) only makes sense when you are writing to a physical PCI-e/Hypertransport/QPI device.

I suspect that this ENA NIC is at least partly implemented in silicon/firmware rather than dom0 s/w.

Perhaps something like Azure's smart NICs? (SIGCOMM 2015 keynote)

20gpbs is fast enough for a raw 4K video stream at 60 frames per second in ABGR format (e.g. what you'd get out of a video card). That's slick.

Tl;dr 20gbps virtual Nic

Oversimplified - it's also a multi-threaded driver with multiple queues. This should hopefully allow you to fully saturate the 20gbps available to you.

Reply on news.ycombinator.com