Linux QoS for humans | Hacker News Reader

Linux QoS for humans

github.com

138 pointsktsaou55 comments

When I used to work on network congestion, I took the position that each stream is entitled to one packet in flight. After that, we can talk about throttling. That's why I invented fair queuing.

TCP will throttle down nicely to one packet in flight, and the connection won't stall, although it may be slow. Routers which discard all the packets from a single stream are doing a very bad job, unless they're dealing with hostile action.

Much of the trouble comes from network devices with big, dumb, FIFO buffers. DSL routers are notorious for this on their uplink side. So are Ethernet interfaces, which have a big FIFO buffer. It's important to avoid too much dumb buffering in the chain between TCP and the Ethernet cable/WiFi air link.

A typical problem is ACKs from downloads being lost during heavy uplink usage. If you're a choke point (much more bandwidth in than out), big dumb FIFO buffers will suck. You at least need fair queuing. The "bufferbloat" crowd is into this.

Linux QOS is more about fixing this as an endpoint. You're trying to avoid overloading the next choke point upstream. This works only if everybody using the next choke point upstream plays nice. One bad guy will ruin it for everybody.

Nagle, much of the issues you're talking about with bad middleboxes is about signaling or optics into their status. We've had a standard for some years to deal with this (ECN). Why don't you think ECN is deployed at all? It seems to me it would be beneficial for all parties.

ECN isn't a solution so much as it is an optional side-effect of routers deploying AQM—the real solution. Widespread deployment of AQM depends on having an algorithm that is both highly effective and requires no tuning or configuration. ECN deployment further requires being confident that broken routers won't cause problems and making a decision of where to put the threshold for marking relative to the threshold for dropping. You want there to be an incentive for endpoints to negotiate ECN, but you don't want there to be too much incentive for endpoints to declare ECN capability and then not respond to marking.

An overloaded middle box can do two basic things - drop packets, or reorder them. Many middle boxes do too much dropping and not enough reordering. Random Early Drop was probably a mistake. It's easy to implement, and beats dropping from either the tail or the head of a dumb queue. But that's its only virtue.

Any packet you drop eventually has to be resent. Packet dropping should be rare. Again, each flow should be allowed at least one packet in flight. This keeps the connection moving forward, and it means the endpoints can measure round-trip time and bandwidth, which should be reasonably stable because the algorithm is deterministic.

If you have so many flows that you can't handle one packet in flight per flow, you're so badly overloaded that no congestion management strategy will work.

Fair queuing is about dropping the newest packet on the flow with the longest queue. Thus, each flow competes with itself, and over-sending penalizes itself.

We really need to get home router/modems, cable headends, and outgoing routers just before the air link for mobile fixed. Those are the big choke points. You can't fix this from the end points alone.

QoS can help with DDoS if the DDoS is L7, but most DDoS attacks are bandwidth exhaustion attacks, which requires upstream mitigation.

I completely agree that "tc" is one of the most confusing CLI programs I have encountered. tc & iptables pushed me away from Linux (for router purposes) to *BSD and pf has been a refreshing experience.

Your are so right! However most of the DDoS attacks I have experienced were amateur, L7 (wordpress, POSTs, etc).

I've really enjoyed pf on OpenBSD over the years. My only frustration has been the ever-changing backwards-incompatible rule syntax which perpetually makes large swaths of documentation useless, including the most recent printed book on pf and numerous online sources.

man pf.conf .. always works :)

Come on, it's not so hard. You mark the packages in the mangle table and you distribute the defined bw with tc.

OK, I confess... all my life I've reused a tc script I inherited from a gentoo sysadmin.

It works, I edit the available bandwitch and magic happens. The iptables cmd is more easy to understand and I even did an output parser and input generators, but tc is hard.

share the magic script. We might learn something.

It's under NDA, but what if I tell you everything networking related are numbers?

It just uses a packet mark convention. There is no magic.

Sounds like a github Gist! I'm also interested in this.

How does FireQOS/FireHOL handle dynamic behavior like interfaces coming/going? My problem isn't so much the tc/ip/iptables syntax or semantics, but stateful mismatch like 'ip route' disappearing when an interface goes down, while 'ip rule' sticks around. Once a config gets that complex it's nice to keep in one place rather than scattered through various files. The easiest hacks are to let rules build up, or to reload everything on any ifup, but ugh. Also, manually allocating numeric table/mark/prio/class numbers is annoying.

iptables (FireHOL) is generally autonomous and does not depend on interfaces being available at activation. tc (FireQOS) requires the interfaces to be up and running. To overcome these issues a) the configuration of both tools is a BASH script itself - do whatever you like in them: variables, conditions, loops, etc and b) in the distribution there is another tool called 'link-balancer' that applies inheritance to routing tables (!) and dynamically alters ip rule filters - it can also manage the balance of gateways.

And all these tools (FireHOL, FireQOS, link-balancer) are just BASH scripts themselves (big ones of course).

FireHOL: https://github.com/firehol/firehol/blob/master/sbin/firehol....

FireQOS: https://github.com/firehol/firehol/blob/master/sbin/fireqos....

Link-Balancer: https://github.com/firehol/firehol/blob/master/sbin/link-bal...

I am personally using tcng, which is a configuration language that already makes tc much more usable.

tcng on $ourceForge: http://tcng.sourceforge.net/

Source code mirror by Debian (if wary of downloading from $ourceForge): https://packages.debian.org/source/wheezy/tcng

Interesting. It seems however that its latest release is 2004 !

No problem, it still works. It just internally calls tc. It's a language that makes it easier to write tc classes and matching rules. The documentation is here: http://linux-ip.net/gl/tcng/

The QoS userspace interface (tc) has not changed much at all, and at least my configuration has no issues.

This looks intriguing, thank you! Over the last 10 years I've tried a number of things to get QoS working on Linux, with not much success. My most successful (and recent) attempt was to purchase a Mikrotik, but to be honest I'm not totally satisfied. This seems like it might be a better solution!

What trouble are you having with MikroTik? I enjoy them.

Mikrotik has been rock solid for me. It's my lack of knowledge really. I adapted someone's QoS configuration and it's been working really well. Much better than anything I've tried before.

My only complaint is the fact is the proprietary nature of Mikrotik. When I was running a small linux box as a router, or when I was running DD-WRT I could always extend the functionality in the ways that's not possible with Mikrotik. The configuration, QoS, everything is Mikrotik specific. I understand that it's reminiscent of enterprise grade hardware though.

How does the QoS interact with containers? Would probably need to configure at the host level, correct?

Wow, that "netdata" daemon/dashboard this is associated with is awesome. I'm wondering if it can be extended for more app metrics, like a custom solution we had at a former employer using grafana.

Regarding QoS and containers, it can be applied at either side (veth is good too).

Regarding netdata, of course it can be extended. Check this: https://github.com/firehol/netdata/wiki/External-Plugins A lot of other users are already writing their plugins - check the github issues

That first paragraph is so right. `tc` is as close to deep magic as anything I've come across in a Linux box.

Project looks extremely interesting. Will have to try it out.

thanks!

previously https://news.ycombinator.com/item?id=11477915

You are right! The title (or the timing) did make a difference though.

I hope your take away is not that it is a good idea to embellish in the titles. There is a reason editorializing in titles is frowned upon; it's a hyperbolic arms race and everyone is a loser.

I am just trying to understand how this works. Probably is the timing. How do you explain the difference in involvement between the 2 posts? I would love to understand what made the difference. Can you explain it?

I cant explain it. What I am concerned about is that you are focusing on it. Would you give your links linkbaity titles if it meant more traffic?

Empirically, yes.

I am an open source developer. I don't sell anything. I am just trying to let people know there is something they can use to make their lives easier.

I don't care about traffic (actually I don't want unrelated traffic - a lot of noise - the key resource wasted is my time), but I do care a lot about involvement.

Since a title can actually trigger involvement, I understand it is good, especially for the readers. Isn't it?

To my understanding, it is good since people that should have been informed about the context in question, have actually been found and informed.

Check this thread for example. A lot of people found something new, learned something. Compare it to the previous one. No one learned anything.

So, what do you think? Shall I pay some attention on titles and their effectiveness?

In Submissions
Please don't do things to make titles stand out, like using uppercase or exclamation points, or adding a parenthetical remark saying how great an article is. It's implicit in submitting something that you think it's important.
...

Otherwise please use the original title, unless it is misleading or linkbait.

https://news.ycombinator.com/newsguidelines.html

I didn't do any of these.

Take also into account, I am the author of the original article too. So, would it be better to copy the old article to a new one with the new title?

You are avoiding the question though: Since a title triggers involvement, is experimentation on the titles something good or bad?

Is it overly pedantic for me to say QoS has nothing to do with inbound traffic or is that an older, say, traditional definition?

My understanding is one can only "shape" outbound traffic. You are under no control at all of the incoming packets.

Linux kernel magic. It is called IFB. Check this explanation at the FireQOS tutorial: https://github.com/firehol/firehol/wiki/FireQOS-Tutorial#con...

You can install an ingress queue and apply a filter to inbound traffic. There are few other more arcane techniques but you're right that control is quite limited.

Here's an example for something I was working on yesterday (traffic policing to simulate slow download):

    sudo tc qdisc add dev eth0 handle ffff: ingress
    sudo tc filter add dev etch0 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate 30kbit burst 3k drop flowid :1

Here's an example of using IFB to shape incoming traffic (what the OP was referencing): http://serverfault.com/a/386791/301799

It's definitely an outdated definition that comes from the mindset that QoS means shaping traffic according to what protocol it uses. That's far too narrow a meaning, and the technique of explicitly classifying traffic is mostly obsolete given the availability of AQMs and FQ that need little or no configuration.

This is awesome. Would be great if it gets OpenWRT integration as well.

Oops, never mind: http://firehol.org/installing/openwrt/

Guess this is too old school ?

http://lartc.org/wondershaper/

Yeah, it's even deprecated by people like Dave Taht (fq_CoDel).

http://www.bufferbloat.net/projects/cerowrt/wiki/Wondershape...

I'm curious -- Have you ever tried it? I could never get it to work.

I never managed to make it work either.

What are the pros/cons of using FireQoS vs. OpenWRT's SQM?

I don't know. I have never used SQM. I guess, FireQOS can work on your servers too. Read the article pointed by this post.

It works well for me. I run it on my local machine to make the network more responsive during large downloads.

sudo wondershaper eth0 7500 850

You'll get better results from

sysctl -w net.core.default_qdisc=fq_codel

on as many machines as possible, and use something like OpenWRT's SQM instead of wondershaper on your router.

Slightly off topic -- is there a solid openwrt router you can recommend that supports 802.11AC?

Congratulations on breaking the trend of "for humans" projects being written in Python. ;-)

I am not that familiar with iptables/tc, but could the tc definitions created by fireqos be consumed by iptables + ipset (say I have an ipaet storing ip addresses and I want to provide a different qos profile for each ip) ?

QoS and Firewall are different things in Linux. I have developed FireHOL (iptables, ipsets) and FireQOS (qos) to handle everything. They cooperate to share for example MARKs, but can also be used independently.

Also featured recently on HN is this nice tool: https://github.com/thombashi/tcconfig

I’d just like to add some other cool QoS-projects to the discussion so that maybe more people can find a project that suits their own needs. I’ve always been fond of QoS, mainly because it was a necessity with five family members and just one measly DSL-connection, second because it’s just neat to use QoS to prevent people from breaking your connection and making everything much more effective.

1. It seems like the first QoS-solution I ever used is still being actively developed. It’s called Mastershaper and you can find it here: https://oss.netshadow.net/redmine/projects/mastershaper Though it seems like the maintainer is doing some heavy work right now so there is no usable version at the moment. In my old days Mastershaper was programmed in PHP and it either worked with iptables as filter or with the faster tc-filters. It had a nice GUI and also some flow analysis through a graph. It was made to run under Debian and the like.

2. The other project I would like to mention is a router firmware called Tomato. The original website might be still available but the original author has ceased development for several years now. It’s only being developed by its community now. You can find the forum here: http://linksysinfo.org/index.php?forums/tomato-firmware.33/ This firmware can be run on several open commercial home-routers (MIPSR1, MIPSR2, ARM), one of them being the old and very popular WRT54GL. But of course it also runs on more recent models. Tomato has several flavours. Toastman’s Tomato is known for its well implemented QoS and stability. You can find it here: http://www.linksysinfo.org/index.php?threads/toastman-mips-a... Tomato probably has one of the most easily usable QoS-systems there is. Once installed, all you need to do is measure your line speed throughout the day, deduce a safety margin of about 15-30% to start with, enter your upstream and downstream values into the GUI, hit “Save” and you are running a pretty decently pre-configured QoS-system. It even supports overhead calculations for ADSL. Unfortunately it doesn’t work with IPv6, yet, and also uses the old netfilter L7-filters, which are pretty outdated. A switch to nDPI hasn’t been undertaken, yet. If you think about using Tomato with QoS, keep in mind that QoS needs a lot of CPU power, so it’s important to know your line speed and buy accordingly.

3. Another firmware that probably has well-made QoS is Gargoyle: https://www.gargoyle-router.com/ I haven’t been using it but maybe it’s worth a look.

Reply on news.ycombinator.com