Hacker News Reader

Comment by ninegunpi | Hacker News Reader

Yes we do, there are problems which beg to be fixed. Most of current complaints are about security, in one way or another require plenty of CPU horsepower (which most routers don't have - CPU peaks while suddenly switching upling and retriving fullview is still a typical picture in a modern ISP). What it takes to prevent something like NLRI spoofing? Plenty of layers of data security.

And it still won't fix the human problem underneath it.

Security-aware BGP versions like S-BGP, soBGP and psBGP address most of security concerns, add plenty of router load and still... don't solve some of the human problems.

But current publicly made suggestions on replacing it with something completely different even lack proper understanding of design goals of original BGP standard.

View on HN

Many years ago I ran a small hosting ISP, ~10 cabinets. One of the choices I made and still stand by was I used Linux-based routers.

We could throw tons of CPU horsepower at it and the fast path and the slow path were the same thing. We had deep Linux expertise, but only passing Cisco expertise, so staying with the Linux networking stack required less specialized knowledge.

But, while most people were worrying about the size of the BGP tables, or trimming the announcements they were receiving, we could get full feeds and never had memory problems. We could run strong filtering, to make sure we weren't sending or receiving any junk like bogons or our users weren't sending with spoofed addresses, without worrying about adding one rule running past the ASIC memory and causing everything to go slow path. We had high availability. All for around $2500 in hardware.

It always seemed crazy to me that they would put such small CPUs in the high end routers. I know they tried to have that CPU do almost nothing, and the ASICs do all the heavy lifting. But for our needs, having an insanely fast CPU and dumb interfaces was perfectly adequate, and let us do full BGP feeds on a $1200 router.

The only real trick we did was we used one network interface and VLANs, and had the multiple network connections terminated at the switch. We were able to get feeds via fiber that we could just terminate in an SPDIF module in the switch. We had 4 core CPUs, so this allowed us plenty of horsepower to manage packet storms or DDoS instead of livelocking the kernel with too many interrupts (it took a while for the kernel to switch from interrupts to polling, and if packets ramped up too quickly the kernel couldn't switch if you didn't have more CPUs than interfaces).

I run a not too small hosting ISP today, and ten years ago I used routers based on commodity hardware running OpenBSD. Back when we were at Gbps scale and most customers were on 100Mb links, this worked fine. However, it did not scale to 10Gbps+, let alone the 100Gbps+ of capacity we're at today.

Beside the obvious case of performance not keeping up in aggregate, performance for just customers being on 1Gbps links was not as good. We found individual TCP session throughput routed through commodity hardware was noticeably lower than using a layer 3 ASICS based switch despite various attempts at tweaking various kernel sysctl variables.

Also, because nobody operates commodity based hardware routers at carrier scale, you cannot trust any open source BGP daemon's implementation of BGP confederations if they even have it at all. At that, I wouldn't really trust any vendor other than Cisco or Juniper for carrier scale routing.

So even if someone were to create a popular, open source routing project making use of DPDK to increase performance scalability on commodity hardware, I'd still stick to Cisco or Juniper solutions like CSR 1000V or vMX for the foreseeable future if I really wanted to use commodity hardware, as the full feature set is going to be proven and mature. I'd want to see any new product stand up well for 5+ years at carrier scale production use before I'd give it any serious consideration.

Even then, I'm not sure how well DPDK will stand up to UDP reflection volumetric type attacks which now dominate DDOS. The top packets/s levels achieved are barely sufficient for simple routing, and would likely drop precipitously with a modest ACL in place, in comparison to ASICS based routers being able to still do full line rate with ACL's that have a large number (hundreds) of terms.

So what is our option for securing BGP-like functionality? The thing that triggered me to tell this story in reply was the part about how most routers don't have the CPU to do the security necessary for more enhanced security.

At the time I was deploying Pentium D in the multi-GHz range routers, the hot-shot Cisco routers were running 100MHz MIPS CPUs, IIRC.

I know they put a ton of emphasis on the ASICs, but seems like they maybe need to splurge a bit on the CPUs. :-) That being said, I haven't really been doing much networking these days, I now just work for a single org, we let the facility do the heavy routing.

Not sure how far back you're going, or which model of Cisco router you're referring to. I first cut my teeth on networking working at a Tier 2 carrier back in 2005 on Cisco GSR's (12000 series) which use a 200MHz R5000 MIPS CPU, but they were already quite long in the tooth at the time and were one of the few remaining networks still running them. And I do recall implementing ACL's to be an issue. Not sure if you were referring to ACL's or BGP policies themselves when referring to securing BGP.

These days, a full BGP capable router starts with the ASR1001 for Cisco which starts with a 32bit 1.5GHz CPU + 4GB RAM on the RP1 and goes up to a 64bit quad-core 2.2GHz CPU + 8-64GB on the RP3. Juniper land, the lowest end non-EOS router would be the MX80 which comes with a 1.33GHz PPC CPU and 2GB RAM, but unofficially Juniper will always steer you towards the MX104 with a 1.8GHz CPU and 4GB of RAM. I can't comment on the ASR's, but we have MX80's and we can do line rate ACL's with hundreds of terms in ASICs without issue. BGP policies themselves are still handled by the CPU though, and are indeed quite slow on the MX80's with full convergence taking up to 20 minutes or so. We've mostly relegated them to core switching duty at this point though, and are waiting for the new MX204's (800Gbps in 1U) to become mature enough before replacing them.

Moving up a bit, we use MX480's as well which we currently have routing engines with 2GHz Intel CPU's and 4GB of RAM, which I believe these are already EOS, but not enough of an issue for us to buy upgraded routing engines for this platform. These do full table re-convergence in about a minute or two. I believe quad-core 1.8GHz is standard now though, with up to hex-core 2GHz available. I don't think CPU's are really much of an issue anymore, although obviously still not as fast as what you can find easily enough in commodity hardware. The ASICs handle most things flawlessly though.

Well... As someone outside of that arm of the industry, I have to wonder about what exposure there is to spectre, or what kind of patches are coming out for affected machines.

JunOS is based on FreeBSD, which doesn't have a fix yet. Not sure about the different variants of Cisco IOS. You wouldn't run untrusted code on a router though, so spectre would not be a concern. In fact, it's probably better not to patch for it given the performance degradation.

That worked fine back then, but in-kernel routing/switching does not scale beyond 1-2 Mpps. Much less if you have complicated iptables rules.

We're getting there with DPDK and similar technologies, but right now, if you're running a high-bandwidth network, there's really no alternative to buying ASIC-based routing equipment from the big vendors. Even more if you're targeted by DDoS attacks, which represent the worst case: tens of gigabits of <100B packets.

Source: work for a SaaS company, we tried it all

I've done mostly the same thing you describe and it absolutely works and works well. You throw much less money into hardware, you don't need to rely on vendor-specific experts and expensive certification training and you don't have to pay out the nose for vendor maintenance agreements and licensing. It also requires that your people have real understanding of how routing and switching protocols actually work rather than being glorified [insert vendor] sales consultants. Add some programming and you have SDN and an incredible amount of power and flexibility with very little siloed, non-adaptable expertise.

Reply on news.ycombinator.com