Hacker News Reader

Comment by zamadatix | Hacker News Reader

It's a MikroTik issue in the end, their buffers are too simply too small for the interface speeds they support. It's part of being budget chips. Flow control is a poor solution as it blocks all flows whenever one flow makes things busy, best is just a properly scaled chip.

I swapped a more expensive "low end" (for the family) Broadcom Trident 3 based switch in in place of 2 different 10G MikroTik boxes and the mixed speed transfers are now behaving exactly like one would expect without any flow control.

View on HN

Yeah, I did get the feeling that Flow Control (while it may alleviate some issues and/or fix others entirely) is just a hack. If what you're saying is true regarding the buffer sizes, I don't understand why they would knowingly make that choice as it causes poor performance which is not what their devices should be known for. It also completely negates the whole point of one 2.5Gbps port in the first place!

I'm curious though: Going by your statement, wouldn't even a large buffer fill eventually? And by eventually, in computer terms, that might be milliseconds/a few seconds if we're talking transfer speeds hundreds of megabytes or gigabytes per second. How big is a buffer supposed to be exactly?

I've replaced several MikroTik 's with something more expensive several times due to just being too slow under load , it's just how they are. You need to spend real $ to get real speed, even for gigabit connections. I know people to use them extensively who say the same.

Packet buffers get a bit complicated but the hand wavy explanation is you want enough buffer to be able to smooth over situations like mixed speed goes-intos and goes-outtas or jitter from small bursty streams but not too much buffer you end up delaying bandwidth/loss discovery algorithms or holding onto packets already considered stale more than you are actually helping. In the end for local switching you generally want a very small buffer but not necessarily just the smallest buffer you can find.

There are endless rabbit holes of details though. You want a buffer with at least 1 hardware queue per network queue (in advanced situations you may even want a hardware queue set per flow) and you want these queues to be intelligent - priority queues, weighted round robin queues, best effort queues. In each of these queues you also probably want things like WRED which counterintuitively helps avoid congestion by starting to drop packets _before_ the buffer is full. You also want the buffer to be as "flat" among interfaces as possible, e.g. if the switch has 12 interfaces it's best it's all 1 line rate ASIC with 1 shared buffer not an 8 port + 4 port with interconnect and it's own set of internal QoS. For non local things latency also starts to become as big a factor in buffer sizes as interface speeds.

I forget what I ended up finding the buffer size of my MikroTik switch to be but I remember thinking it should have been 4x-8x the size - whatever it was it was truly tiny and didn't seem to be operating well.

As for why MikroTik makes things with known too small buffers... it's cheaper than proper high speed interfaces but more performant than just using low speed interfaces. Sure, it's not a perfect 2.5 Gbps solid but for the price it'd take for them to do that "right" they would be able to do 10 Gbps ports "wrong" instead. I still use my MikroTiks though I just make sure not to mix interface speeds or if I do it's with something on the side where I don't really care about having perfect performance to it just a generally fast connection when I do use it.

"goes-intos and goes-outtas" - love it.

Buffer counts and buffer sizing is also a notoriously cut-throat game in vendor marketing, almost on par with (mis-)use of the term "wire speed".

This is not an easy/cheap problem to solve, and I fully agree with the conclusion: Mikrotik hardware is good up to the point where the price/performance curve falls the wrong way in a particular situation.

I also get a lot of use from Mikrotik hardware, particularly the passively-cooled models where performance isn't as important as price + silence.

I had to fight this battle in 1998 with a DSL SoHo modem/NAT router (single box) with a 10Mb/S Ethernet jack routing through an ADSL modem with a maximum speed of 8Mb/S Down and 1Mb/S Up. The 'Up' path was killer with the PC trying to stream 10Mb/S Ethernet into a NAT router that could (at best) dump 1Mb/S up through DSL. So a 10/1 ratio.

And yes, make your buffers too big (we had relatively tons of memory) and all you get is weird latency cycles as the higher level protocols over-send, then fall back. Make your buffers to small, and the total throughput falls off, bu the latency is far more 'normal'. We ended up buffer scaling based on what the upstream connection really achieved, so larger buffers for a 1Mb/S link than for a 128Kb/S link. We would start the router at 1Mb/S buffer sizing, and scale down if the DSL link throughput was less than that.

At the time we were working through that, a number of people would scream 'just give it more buffer', not understanding how badly that broke something like TCP.

Isn't this the kind of problem that ECN is supposed to solve? Unless the buffers are very small compared to the port throughput, the upper layers should be able to handle it.

ECN purely allows the sending CCA to modulate sending rate before packet loss. Buffers filling up does not change, what changes is how early the sender reduces their window sizes and whether packet loss occurs before then.

The ECN bit can be flipped by routers but not switches. Layer 2 performance pathologies can’t be solved at layer 3.

In the age of bufferbloat their buffers are too small? How many 1500-byte packets are we talking about, if it's that bad?

Unless overly large buffers are covering up the real problems with TCP?

At 2.5 Gbps having 5 ms of buffer requires >1 MB for that single hardware queue which was either at or over the available buffer for the entire switch IIRC.

Having too large of a buffer, particularly on gigabit class software routers/nat points which use CPU+RAM instead of expensive hardware, is definitely the more common problem. There it's too easy for the trap "I have 512 MB of RAM, why shouldn't I let as much as is available be used as buffer, it's helping" which then makes you wish you had 4 MB of buffer instead.

I noted elsewhere, I fought this battle once. People outside the development team kept saying "it needs more buffers", but they were not looking at the network analyzer, and they weren't looking at the internal states. As buffers became excessive, it started with weird latency bouncing, and it just got worse and worse as you added buffers. We had a network impairment simulator. It was fun to turn that knobs on that and see how that interacted with packet loss buried in your buffer.

Why would you need 5ms of buffer?

Reply on news.ycombinator.com