Achieving 100Gbps intrusion prevention on a single server (2020) | Hacker News Reader

Achieving 100Gbps intrusion prevention on a single server (2020)

blog.acolyer.org

92 pointsharporoeder38 comments

"The slow path can’t take advantage of constant time operations, but fortunately is less often used as most packets arrive in order. It’s also used when inserting new flows."

Going to be in a world of hurt once a slow-path DDoS packets come rolling in.

I really only operate on L7 most of the time, so pardon me if this is a dumb question: is there some form of TCP rate limiting that could happen at the switch level? WAF? Can policy be applied for the slow path?

Is there a way to protect against that type of DDOS you raised?

Switches don't care about TCP.

You are wrong.

https://en.wikipedia.org/wiki/Multilayer_switch

See for an intro.

Respectfully, you are incorrect. Switches, in the technical sense and accepted terminology, refers to Layer 2 processing. Routing is Layer 3. And so on. MLS is just a combined product offering multiple Layers of processing.

I think this argument can never be won. "Layer 3 switch" is common terminology. But "switching" strictly speaking is a Layer 2 action. But sometimes we say that a switch is "switching packets at Layer 3" when it is doing a hardware action in response to IP layer information. We could go back and forth all day. So let's all be reasonable if possible.

I believe “L3 switch” and routers(L3-L7) are distinguished by architecture; L2 and L3 switches employ non-programmable “packet switching fabric” ASIC with CPU acting as a control system, while routers are generally a general purpose computer optionally with non-Turing-complete ASICs for faster packet processing.

Expectations of a “switch” is therefore that it’s not a dual core PowerPC box with 24-96 GbE ports on PCIe, running outdated Linux Kernel, and that it can’t do what such a bare metal box could do.

I'm not sure what is non-programmable about the packet switching fabric. It can be programmed to switch packets, which is what we want the device to do. It can also be programmed to route packets, which is done at the same rate as switching them (usually line rate). So we can call this "layer 3 switching" because it is the same process as the L2 switching but it is happening at L3. That's what a L3 switch is and does.

It can't do the same as a box with general purpose CPU, but it can do the thing you bought it for (routing) at line rate (hence the comparison to switching).

https://www.cisco.com/c/en/us/products/collateral/switches/b...

There’s more to switching than Bestbuy home equipment. I was going to say Netgear, but even they offer some layer 3 switches I think.

I'm painfully aware.

My Brocade FCX 648S-HPOE arrived from eBay yesterday. See I have a homelab setup I'm cobbling together and a mission to train a door to recognize and block my neighbor's cat from entry. Her name is Aria and she pisses everywhere then eats the cat food. I have 3 cats that require free use of the cat door, and if its closed they piss everywhere.

I've been scheming about how to do this for quite some time. The basic idea would be to install a magnetic lock on the cat door, and actuate it over an MTQQ triggered relay. But how to trigger it? My cats refuse to wear collars and their microchips weren't readable within usable proximity. Enter https://frigate.video/ this summer. Its a self-hosted NVR that can be trained to recognize arbitrary objects and fire off events when objects are detected, including to MQTT. It looked like a viable project, and I've been trying to get some camera system anyways for minding the front door while I work from a distant basement- but I haven't been willing to join the Ring panopticon just yet.

Over the past few months I've been acquiring the required hardware from eBay. I overpaid for a Google Coral USB TPU, and got a steal on a pair of their recommended cameras, Loryta IPC-T5442TM-AS-LED unused from a commercial install job. Unfortunately they are POE only, or a propriety 12VDC. I know I was going to need POE eventually anyhow, and while my Mikrotik RB4011iGS+5HacQ2HnD has a single POE port I would need more - and I wasn't able to get even that port working for one reason or another. So I found a Brocade FCX 648S-HPOE for $50. Overkill? Most definitely. I thought there would be no harm, and it would give me an opportunity to work with serious gear and improve my networking acumen. It is as loud as a laundry machine I swear.

Unfortunately its so serious that I need to go find an RS-232 cable to enable web management - until then it drops all links. So I still haven't been able even fire up the cameras. If my foraging through the cable bins again proves fruitless, then I'm going to their drive around town or find one online and wait until the next weekend...

So that Best Buy home equipment sounds kinda nice right now.

Sincerely,

Pissed On && Pissed Off

Doesn't that switch have an out-of-band Ethernet port on the back? It does according to the manual... Try it?

https://www.manualsdir.com/manuals/361627/brocade-fcx-series...

The Brocade FCX 648S-HPOE has is a stackable switch with forty four 10/100/1000 Mbps ports plus four Combo ports, which include four 10/100/1000 Mbps RJ45 ports and four 100/1000 Mbps SFP ports. The switch has two management interfaces, a DB9 serial port (Console) on the front panel and an RJ45 port (Out-of-band Management Interface) on the rear panel

A layer 3 switch does not just “glean” information from the packet , it can switch packets and rewrite IP header data at wire speed to place packets on different networks completely bypassing a router. I don’t know of any better term for it than a “layer 3 switch”.

But these are not “straightforward” concepts - you say there is no “hybrid” thing - but there most certainly is: https://en.wikipedia.org/wiki/Bridge_router

From reading that I see L3 switching uses "specialized ASICs with the help of content-addressable memory" and later on features "flow accounting". I'm supposing those primitives could make a rate limiter. I'm refusing to go down this rabbit hole though. If I'm wrong I'm sure I'll hear about it by tomorrow :)

Yes, switches can do this with policing. The hardware can inspect into higher layers in (certain, specific) ways.

Switches (and other network devices) can cause TCP senders to rate limit their transmissions, and can do so without even being aware of TCP per se. This[0] page does a good job of explaining it in detail, but the gist of it is that TCP implementations try to match their transmission rate to the capacity of the underlying network by maintaining a congestion window: a sender starts sending slowly, gradually ramping up its transmit rate as long as no packet loss is observed, but quickly and drastically reduces its transmit rate when packet loss occurs.

This is behavior is manipulated by non-targeted congestion control schemes like Random Early Detect and targeted schemes like traffic policing and shaping.

Beyond consuming slow path resources, you might also be interested in SYN cookies to mitigate state-table exhaustion attacks.

Note that UDP, by itself, does not respond to loss in the same way. Any congestion control would have to be implemented at the application level, although policing / shaping are frequently applied to UDP traffic with good effect.

One minor nitpick, though: these tools are generally, but not always, effective at mitigating single-source DoS attacks. For volumetric DDoS, they don’t really work at all. For those, you need a provider, like Cloudflare or Akamai who can divert the malicious traffic away from your network. By the time the end network receives those packets, the damage is almost always done - it really doesn’t matter if you drop or forward at that point, if 100% of your Internet connection is filled with malicious traffic.

[0] - https://witestlab.poly.edu/blog/tcp-congestion-control-basic...

This seems like a whole pattern of attacks that can be defeated with a single counter and random-early-drop.

Nope. Need an array of counters, a big big array.

Because...

You want people to know how to generate these slow-path DDoS, do ya?

Been there, on both ends, ... within a network accelerator lab.

I spent several years as the lead on what was at the time the most widely deployed DDOS product in the core of the Internet; I do not know what you are talking about. Could you maybe be more specific about why this particular DOS attack requires extreme statekeeping from middleboxes? I don't think it does.

Like. I said, within a network accelerator lab (or in laymen terms, an artificial DDoS whitelab environment for testing middleboxes).

“with hundreds of thousands of concurrent flows ”

Isn't an IDS meant to be used in a LAN though? If you can craft a packet that special and hit this LAN device with it, wouldn't they have _way_ bigger problems than a dos?

If you need 100Gbps of IDS you probably are running that in front of something internet facing.

Large corporate or college campus?

This is a cool idea like the potato powered clock. There are so many holes here. Let me just pick one. They don't seem to account for reassembly issues which is a huge problem and vastly multiplies your problem space depending on how you implement the solution. What the fuck am I going on about you ask?

Think of it like this: sig: abc Traffic a[1] b[2] c[3]

where the packets are properly ordered in 1 2 3 order. Simple fragmentation could be sending them out of order - I believe this paper accounts for that. What if instead you send a[1] b[2] b[2] c[3]? Windows assembles this one way (depending on the version), linux another, bsd another. It's super fun. Then what if you send c[3] b[2] c[3] a[1] b[2]. One could argue, "hey d*ckhead we're going to normalize the traffic first" the problem is what is normal? Stevens had tons of good work on this. Some systems have a 'normalization' standard that's similar to how their network gear works. Also I find the fact that they say 'all the patterns' must be matched for the sig to fire. Does that include an or? Are they breaking the or down into sub detectors or something? The 10,000 signature thing is also kind of fake as the number of signatures constantly grows like the number of amazing taylor swift songs.

All in all these authors need to go read the old breakingpoint test standards, or ixia, or nss, or really anyone.

The article does in fact mention both reassembly and out-of-order packets.

Reassembly and ordering is one of the major focus points of this research. They talk very much about this and the performance implications. You can literally "ctrl + f 'reassembly'" ??

This all has very real world applications like with Corelight.

My point was they only address the most simplistic case.

> One could argue, "hey d*ckhead we're going to normalize the traffic first" the problem is what is normal?

This is why this kind of IPS integrates with a firewall well. Two decades back, my team built very fast for its day firewall that would only let assembled-and-refragmented fragments through.

There was no confused ordering past the firewall, and no scenario of IDS/IPS and victim defragmenting differently.

For multi-server situations, using Arista's Tap Aggregation (TapAgg) architecture is a popular solution for splitting up network packets after doing a optical fibre tap on the network connection:

* https://www.arista.com/en/solutions/tap-aggregation-with-dan...

The hosts would run software like Suricata, which is multi-threaded, and so can take advantage of many cores. (Until recently (3.0?) Snort was single-threaded.)

Discussed at the time:

Achieving 100Gbps intrusion prevention on a single server - https://news.ycombinator.com/item?id=25108392 - Nov 2020 (35 comments)

I’m yet to see a signature matching system with defaults which are actually good. Don’t get me wrong, the ability to add regex filters are infinitely valuable for custom or emergency mitigations, but I think largely current systems blocking heuristics which look like attacks aren’t solving much.

That being said, I’m interested to see anomaly detection engines that learn data patterns and flag potentially malicious traffic.

I don't believe reasonable defaults are possible for that use case. I mean, some anomalies are sure interesting. But I don't care about people trying to exploit MS exchange on my Linux boxes... Everyone runs something different, so unless a lot of profiles are provided that you can turn on/off, what would a reasonable default even be?

That’s my point. Though, maybe there is something to be said for blocking things like OWA vulnerabilities as compared to generic xss/sqli detection rules.

General purpose CPUs seem to have peaked - purpose built hardware is definitely on the rise. It worked well for GPUs (FPUs used to be external too) so I guess it shouldn't be a surprise. Mainframe on the desktop!

Direct link to the PDF https://www.usenix.org/system/files/osdi20-zhao_zhipeng.pdf (not suggesting skipping the intro post as it had some good context/color)

For completeness, this is a (very good) November 2020 (two years ago) write up on a paper [1] with a video presentation (19 minutes) [2].

[1] https://www.usenix.org/system/files/osdi20-zhao_zhipeng.pdf

[2] https://www.youtube.com/watch?v=KCUCUMNPMl0

Reply on news.ycombinator.com