Hacker News Reader

Comment by jiggawatts | Hacker News Reader

The most important and most useful thing is the hardest to obtain: latency simulators.

I just want an IPv4/IPv6 proxy that does nothing other than delay, rate-limit, and/or drop packets.

This is a critical aspect of application performance that all developers just ignore because it's such a PITA to simulate.

Your loopback adapter has a latency of 30 microseconds, but that same web-to-database connection might traverse multiple switches, routers, firewalls, and load balancers and have a latency of over 1,000 microseconds (1ms) in production. That's thirty times worse, and I'm not even talking about zone-redundant deployments, regional-to-region failover, or any such thing!

Ditto with web apps, where every developer mistakenly believes that their Angular or React monstrosity is "performant"... because it is... on their own machine. With loopback networking. Try that over a 3G connection... in rural India. Not Indiana. India.

View on HN

> I just want an IPv4/IPv6 proxy that does nothing other than delay, rate-limit, and/or drop packets.

Built-in into linux kernel, althought it's per interface unless you get dirty with traffic shaping tools.

Also you probably dont want proxy as support for proxy servers can be pretty iffy across software stack.

> Your loopback adapter has a latency of 30 microseconds, but that same web-to-database connection might traverse multiple switches, routers, firewalls, and load balancers and have a latency of over 1,000 microseconds (1ms) in production. That's thirty times worse, and I'm not even talking about zone-redundant deployments, regional-to-region failover, or any such thing!

Reminds me of a time when COVID hit and the suddenly-remote developers were suprised the time to start of their application went from 5 to 30 minutes (they connected their app to DB server on dev environment to not carry big-ass database with them)...

> Ditto with web apps, where every developer mistakenly believes that their Angular or React monstrosity is "performant"... because it is... on their own machine. With loopback networking. Try that over a 3G connection... in rural India. Not Indiana. India.

And that reminds me of this time frontend developers loading 700MB of data to load the site and not noticing that, but sending ticket to ops "server is a bit slow, could you give it more cores?". Said developers were working in office that had direct connection to data center 500m away with sub-ms latency and plenty of bandwidth.

I had many times pondered traffic shaping frontend devs connection to like 10Mbit after that... maybe 56kbit for repeat offenders.

> I just want an IPv4/IPv6 proxy that does nothing other than delay, rate-limit, and/or drop packets.

About 15 years ago I needed to emulate a network over a satellite link. We had limited amount of time on the dish and it was a fairly costly affair. We had a small rack of hardware together with a bunch of measuring instruments that would be in the field for data acquisition. It would be sending back data home where it would be processed and then sent back into the field. The (limited) bandwidth reserved and the inherent latency on the link gave us some interesting issues to deal with, but it required a few iterations to get things working smoothly.

The rack had a Linux box which acted as a router (among other things), and while it was in the office we'd just hook it up via ethernet. So I used tc[0] on there to introduce a fixed latency on transmission and cap the outgoing bandwith to whatever was available on the link. I did the same on the homestation for outgoing traffic but there I just used an old box with two ethernet ports and set it up as a bridge.

For dropping packets I used iptables, and some other things.

It requires some familiarity with the LARTC[1], which isn't the most readable document, and how things work in Linux. It gave us exactly what we needed without having to pay for time on the dish for testing.

It's been over a decade and the details are extremely vague, but I'm sure that if you want to you could mark certain packets with netfilter and then delay those packets somehow, rather than just delay the entire network device. I remember that with iptables we used to mark packets from SSH in the mangle table and then use that mark with tc to give traffic marked priority on the outgoing device.

It's not exactly a proxy, but back in the day it worked for that specific use case.

[0] : https://linux.die.net/man/8/tc

[1] : https://tldp.org/HOWTO/Adv-Routing-HOWTO/

tc is the way to go if the options are sufficient. You can bridge two interfaces (also virtual ones) in a VM and set qdiscs. Also works asynchronously, e.g., different rules for up/downlink. Instead of a VM setup one could use a cheap linux/openWrt router or an SBC-type board.

Tc is always a pain to search for, I add 'netem' as a recommendation for keywords and also the IFB trick to put a qdisc on an input stream.

Linux's tc-netem [0] is designed for this:

"The netem queue discipline provides Network Emulation functionality for testing protocols by emulating the properties of real-world networks.

The queue discipline provides one or more network impairments to packets such as: delay, loss, duplication, and packet corruption."

[0] https://man7.org/linux/man-pages/man8/tc-netem.8.html

tc-netem is too complicated to set up and its performance profile is very different to real networks.

I mean this in the sense that it can’t even emulate a “long cable” properly.

I’ve tried both the Linux and BSD tools mentioned by others here and they’re all very old, weird, and difficult.

Huh? FreeBSD had this 15 years ago already (if not more). Thingie is called DUMMYNET. Imunes utilizes netgraph thingie that can be connected thro pipes (DUMMYNET) to emulate whatever delay, bandwidth and BER you want. Very handy thing to check out how your stuff will behave on dialup for example.

This is my BGP troll lab for some testing ;)

http://ds-1.ovh.uu3.net/~borg/pics/bgp-lab.png

There is clumsy which simulates bad network http://jagt.github.io/clumsy/

And comcast: https://github.com/tylertreat/comcast

This looks great! Besides that it seems to be only for Windows and requiring a GUI to configure the values...

Anyone know of something similar for Linux that can be configured via config/env vars/cli?

there's labrea which is pretty dusty: https://github.com/dustin/labrea

you could probably achieve something similar with bpf today.

IPFW dummy net in FreeBSD was amazing for this. I haven't used it in 20 years. On my home DSL line in the 90s I would get so many hack attempts that it was becoming a nuisance, so I filtered ICMP through a dummy net that both delayed and dropped 10% of the packets. Hack attempts went to zero in 3 days.

Luigi Rizzo is a top notch hacker.

https://github.com/luigirizzo/dummynet

I suppose any proxy can be used in combination with https://www.linux.org/docs/man8/tc-netem.html but it is a bit tedious.

Perhaps one of the linked projects can help with it?

Does 'tc' fit your need? its a simple solution to that sort of need: https://man7.org/linux/man-pages/man8/tc.8.html

I use this to simulate delays between various local services:

https://github.com/Shopify/toxiproxy

If you have Docker all you need is a few terminal commands

For web apps, chrome has a throttling tool in the dev tools. Not sure how sophisticated it is, but I've used occasionally to recreate bugs only noticed by clients on slower connections.

On Cloud you could look at the AWS Fault Injection Simulator. This example is in the context of EKS but you could use it for EC2.

"AWS Fault Injection Simulator supports chaos engineering experiments on Amazon EKS Pods" - https://aws.amazon.com/blogs/containers/aws-fault-injection-...

Your proxy adding latency is a very good idea. I'm wondering what it would be wrote in. Go, so you can integrate it easily on a k8s as a side container or else ? Rust, so it can be performant and at low level ? Nim, so it can be easy to write and performant ? I think that can't be that difficult to write, but maybe I'm wrong. I will look into it but I don't promise anything. I think it is a good pet/side project.

You can also write it in Python, so it can be slow while it slows down your packets.

Ahah good one !

https://github.com/tylertreat/Comcast

I’ve noticed that Hacker News is a lot faster than other sites and apps on a bad connection probably because it’s just basic HTML

Basic HTML isn’t static HTML. The comments and posts for Hacker News still needs to be stored and loaded from a database.

That doesn't matter for the client through. As long as the data can come in a single small file, for example the <3KB HTML file for the tab I'm writing this on, what the server is doing doesn't matter.

Well, kind of, at least considering the last public version of Arc, that HN uses (found here: http://arclanguage.org/)

It seems to be storing stuff directly on disk, on the same host that the software itself runs on.

So you're right if you consider the filesystem a sort of database, but otherwise no :)

It was announced in 2008! 15% into the 100 year language.

Not sure their tech stack (I'm sure there's a post on it somewhere) but there was a forum software way back that would compile replies and such into static html pages when the submit button was hit, on the assumption that they would be read a lot more times than written to. Worked pretty well from what I recall.

Great idea in theory until you want to change the presentation and now have to recompile 3̵7̵8̵4̵3̵1̵0̵5 (37843314) items :)

True, but it was split into pages so at most 20 items :)

That doesn’t happen over my mobile data connection so I don’t see how that’s relevant to my point

Afaik hacker news pages are stored on the file system.

Because not every letter is a 500 line, 30 dependency component, in other words

Reply on news.ycombinator.com