Comcast: Simulating shitty network connections so you can build better systems | Hacker News Reader

Comcast: Simulating shitty network connections so you can build better systems

github.com

330 pointsot80 comments

Nice.

It'd be neat to include options to simulate other pathological conditions often encountered in the field like:

- Multiple layers of NAT - overlapping un-synchronized timeouts are one thing ... but the REAL fun comes in when intermediate layers have the same IP ranges as IPs you are trying to reach on the "outside" of the NAT sandwich. All kinds of "interesting" things can happen, like the "software laser": http://catb.org/jargon/html/S/software-laser.html

- Stateful NATs/firewalls with bizarrely short connection / UDP association timeouts or that randomly forget connection state

- Shitty NATs/routers that go into bizarre failure modes when there are too many open connections, like forgetting early ones because they remember connections via a small ring buffer

- People who block all ICMP "because security."

- NAT + short DHCP leases = musical external IP addresses

- Small MTUs (<1500) like those imposed by PPPoE and other nasty encapsulation protocols, sometimes without ICMP errors for big packets because some jackhole blocked ICMP "because security."

All these things are common in the field. WHYWHYWHYWHYWHY

My favorite enterprise network pathologies:

- Randomly block specific ports "because security"

- Limit RPC port ranges to some random small number.

- Strange network optimizations. Example: Optimize TCP Window Size to support NT4 clients on 56k frame relay circuits.

- Very Slow DNS response

- Optimize core switching rules to fully utilize switch CPU. Avoid configurations that take place in an asic.

- Long DHCP Leases + Undersized DHCP scopes.

I also forgot TCP through TCP tunnels, which causes strange timing and flow control behaviors due to never actually dropping packets on congestion and double-ACKing.

It makes me wonder at how Skype manages to tunnel though almost anything.

Primer I wrote a while back: https://www.zerotier.com/blog/?p=226

Skype has abandoned P2P, though I'm not sure why. One possibility is that MS now has middle boxes deployed at so many interchange and peering points that there's no benefit in maintaining the added complexity. Another is that it was to comply with surveillance/tapping requirements.

IIRC they said that it's because of the shift to mobile. You can't really do P2P when the majority of your clients sit on phones.

EDIT: [I can't reply to the comment below, so I'll add here] It's most likely not a technological problem, but rather than data usage is limited on mobile and you don't want your user to pay for traffic they didn't use.

I keep hearing a categorical "you can't do P2P on mobile." I intend to fling myself at this problem like a drunk seagull soon, so I'll blog on it. I think it's possible, but with a number of special considerations around wakeup-quantization and general power management, some protocol augmentation, and a high tolerance for nodes appearing and disappearing.

Primer concept: http://en.wikipedia.org/wiki/Interrupt_coalescing

The problem isn't power (well it is; but that can be overcome), it's spectrum and pricing.

Pricing: Most people in the US are charged for the data they use on their mobile devices, and thus would not want P2P used on their phone because it costs them money.

Spectrum: P2P is not a very efficient distribution model in a world where most clients are on asynchronous connections. Asynchronous connections exist because transmission spectrum is limited, so to maximize spectrum usage, telcos allocate more spectrum for downstream transmissions than upstream. But if everyone's phone is chattering all the time with P2P traffic, you're going to saturate the spectrum and reduce overall data speeds. This is why mobile is still charged on a usage basis: it discourages overly chatty applications.

Good point on spectrum.

Verizon and AT&T monitor for super-chatty apps and in some cases can rate-limit or otherwise impair them.

P2P doesn't necessarily mean cooperative relaying or swarm distribution like BitTorrent. I agree that those applications are mobile-unfriendly with current batteries and cell networks. It just means you are talking directly to your peers instead of back-hauling to the cloud. You can have P2P where the only traffic you handle is your own. In that scenario total aggregate bandwidth shouldn't be that different from a cloud-backhauled app -- the only difference you'd see is in how many endpoints you're talking to. So instead of seeing 25mb transferred to/from one IP, you'd see it sprayed across a few dozen IPs.

The big hurdles I see are (in no particular order):

- Connection maintenance and keep alive. Basically this is bad, so you want to be more aggressive about shutting down unnecessary P2P links on mobile than you need to on desktop/server. Keep alive requirements generally suck anyway, and are one way NAT murders kittens.

- Restrictions around background tasks on mobile OSes (iOS is particularly onerous).

- Squelching inbound traffic from badly behaved or broken peers to avoid inbound flooding.

- Battery life and related concerns.

I understand it this way: cellular networks currently have their own "netiquette." It includes things like don't be too chatty, try to coalesce instead of spewing packets at excessively random times, etc. These things are less important on wired networks since they don't have the same resource constraints or bandwidth issues.

I guess a related question is why you would do P2P and not backhaul to the cloud? I can think of many:

- Reduced latency for things like AR and VR where latency matters a lot.

- Reduced bandwidth cost due to lack of back-haul, and back-hauling a pic being sent between two people in the same city 2000 miles to a cloud server is just offensively stupid anyway.

- Privacy and security.

- With P2P you could have a more open app model where apps aren't wedded to proprietary cloud infrastructure. They still work even without someone's cloud, etc.

If you're talking about P2P over cellular, I think it's just in general a bad idea. The way cellular networks are operated, you can't do device-to-device connections - they have to go through the tower for a number of (very good) reasons. That's where the latency comes in, and the wireless transmission latency can be an order of magnitude higher than the latency on a cross-country connection. Furthermore, the resource that is constrained is the amount of available spectrum - so we need to optimize for that. Bandwidth is effectively infinite from the tower to the cloud, but bandwidth from the tower to the device is the constraint.

P2P over non-cellular (i.e. Wi-Fi or Bluetooth) where you can actually make a direct device-to-device connection seems like it may have some use cases (messaging apps, etc.) But they're edge cases and by no means a common use case because it just isn't reliable enough.

Alex Pankratov, the developer of Hamachi, discusses his approach here: https://swapped.cc/#!/hamachi

What do you mean randomly block specific ports because "security"? All ports should be denied access except ports which have a justified business reason. Got a web app? The only thing open should be 80/443. There's no reason for SMTP to be open on the web server. Anything doing mail should be on its own MTA server. Least functionality per server. That's not even security. That's just good system administration.

At one of the enterprises that I've had the pleasure to work at, the network guys would randomly come up with some "concerns" about your firewall requests, and would just not include certain parts of your request.

So you might request ports 4000-4100, and find that 4007 is blocked, "because security".

I'm pretty sure the reality was that the firewall rules were a big hairball, and they were stepping in some other rule out in place a long time ago.

Don't forget jitter, and simulating an ISP that will shut off your DNS traffic at random (literally anything DNS, not just their own servers).

I forgot about DNS munging shenanigans. Not only do some people selectively block DNS, but there are also cases where middle boxes and proxies pretend to be remote DNS servers but really aren't... or rewrite DNS traffic, insert fields, etc.

I take it you've seen core switches on fire off the Jupiter gate?

Juniper gate.

Juniper leads to gin?

Tannhauser gate

>WHYWHYWHYWHYWHY

Are there any network administrators who don't consider NAT traversal to be a security breach? Making it difficult would seem to be a feature, not a bug, in most enterprises.

NAT traversal is a security breach that became a de-facto standard and is part of the SIP, STUN, and IPSec (extended) standards among other RFCs. It exists because NAT itself is an abomination.

If you're on Mac OS, you can also install the Network Link Conditioner from the Apple dev tools, which is a prefpane that serves the same purpose:

http://nshipster.com/network-link-conditioner/

And on the command line, you can use the dummynet(4) command (can be useful to automate testing under various network conditions).

There is also clumsy with what possibly is the best demo gif ever made.

https://jagt.github.io/clumsy/

Tamper mode was downright hilarious.

Regardless of how good the tool is - that was an excellent demonstration clip! Within ~30 seconds I knew exactly what the tool did, why it was cool, and what I might want to use it for.

Startup landing pages could learn a lot from that single gif.

People talk about "fuck you money," meaning the amount of money you need to make before you can never work again. Personally, my "fuck you money" isn't money per se, but the fall of Comcast. When I build enough value in the world to displace Comcast, I will retire.

We need to see more work in the areas of mesh networking, layer-3 routing, and consumer networking in general. This tool is a good step. Personally I'm hacking on some openwrt routers right now -- I recommend everyone try it. The documentation is dense, with a friendly community writing it.

You know your company sucks when its name has become synonymous with high latency, dropped packets and a all round shitty network.

Unfortunately i suspect they will just send a trademark claim to Github to get it removed.

This seems like satire to me. Could that save it like "Dumb Starbucks"?

https://en.wikipedia.org/wiki/Dumb_Starbucks

I don't think Dumb Starbucks was saved. They just didn't ride it out long enough to find out. Who knows what would have really happened though

The problem is that fair use and satire are defenses that must be argued. They don't automatically stop you from being sued. It can results in a lot of upfront costs and hoops to jump through before you are actually granted your fair use exception. You have to be pretty committed to your joke to actually fight something like this to the end.

Although if I were Comcast, I would ignore this entirely. I wouldn't want to risk the Streisand Effect bringing attention to this type of thing.

You can't ignore a trademark violation and keep your trademark. Trademarks can become "generic" (think Kleenex or Band-Aid) if not enforced, which makes them unenforceable. So no, they won't ignore it.

I hate comcast customer service with a passion, and their pricing models are super hostile to existing customers.

But their internet service has been very reliable and very fast for several years for me. I rarely have internet outages.

Just yesterday I noticed my bill had gone up to $147. I called and did the yearly "I'll cancel if you don't give me a discount" nonsense. An hour out of my day.

Btw, they had raised the price of my 5 static IPs to $24.95 (from $9.95). I have to lease their modem ($12.95) because of the static IPs. I did research and found out that my registrar (namecheap.com) has free dynamic dns, so I'm going to switch to that and give up my static IPs.

So, after the $40 discount I got yesterday, plus the extra $38 for the modem/IPs, I should be down to a reasonable rate again.

I fucking hate Comcast.

> I have to lease their modem ($12.95) because of the static IPs.

This isn't actually true, assuming you are on the consumer side of the fence. No experience on a business line though.

Source: Has extra IPs on his owned modem, after fighting with them for a couple weeks.

It is true, what I said. I have Business Internet and they will not allow activation of a non-Comcast-owned modem when you have static IPs. I know, I tried to do it twice.

Ah yeah, business side I have no experience with. Just wanted to make sure anyone buying on the consumer side knows this, as they tried to pull the same crap with me when it was not true.

You can't get static IPs on the residential side, btw.

You just reminded me that I have been dragging my feet upgrading my two OpenWRT routers for the last few weeks. I guess I know what I am spending my lunch break on!

You really shouldn't do something like this without at the very least mentioning Kyle Kingsbery and the Jepesen test suite he wrote to validate how distributed datastores actually follow CAP or not.

For any devops / sysadmin / systems engineer: I highly suggest reading this if not only to understand failure conditions better:

https://aphyr.com/tags/jepsen

The entire "Call Me Maybe" blog series is because that is Carly Rae Jepsen's stupid pop song, for which he named his test suite after. That and Kyle's posts are absolutely hilarious to read. Seriously, read them.

Would something like this be DMCA'd or taken down because of the name? (Not that it is violating copyright by its contents)

I had that thought as well. From poking around various IP law sites, DMCA is purely for copyright infringement, and can possibly open you up to more liability if you try to use it for trademark claims (what this would be), see http://www.lexology.com/library/detail.aspx?g=13f9814f-b56e-... for an example.

If Comcast wanted to take this down, it would be through a trademark infringement claim.

(IANAL, etc, etc)

So... the project is infringing on Comcast's ownership of shitty networks?

The DMCA doesn't apply to trademarks. This probably wouldn't violate Comcast's trademark anyway, as they operate in different markets.

Networking and infuriating call centres?

I wonder if the author can invoke the freedom of satire.

Technically, no. However, it wouldn't be the first (or last) time that a DMCA takedown was issued for the wrong reasons. I can't remember specifics, but I seem to recall DMCAs issued for name conflicts before.

I was confused if this was something from the real Comcast, so it seems to be a clear case of trademark infringement (but it's not my call, and I may be wrong).

If GitHub agrees, then it would be a violation of their Terms of Service [1], section A.8:

"You may not use the Service for any illegal or unauthorized purpose. You must not, in the use of the Service, violate any laws in your jurisdiction (including but not limited to copyright or trademark laws)."

[1] https://help.github.com/articles/github-terms-of-service/

On linux, tc can do similar things, such as dropping a percentage of packets or adding random delays.

http://linux.die.net/man/8/tc

As said in the readme:

> you can use tc which supports some additional options.

     $ tc qdisc add dev eth0 root netem delay 50ms 20ms distribution normal
     $ tc qdisc change dev eth0 root netem reorder 0.02 duplicate 0.05 corrupt 0.01

> To reset:

    $ tc qdisc del dev eth0 root netem

Yeah, I used to do stuff like this with tc directly back in the day, but I could never remember the options. This wrapper seems to make it a bit more obvious what goes where.

From TFA :)

> On Linux, we use iptables and tc. Comcast is merely a thin wrapper around these controls.

yep, https://github.com/tylertreat/Comcast/blob/master/throttler/...

I don't need to simulate this, I have actual Comcast service.

I don't need this, I have comcast.

    $ sudo ping -i 0.02 8.8.8.8
    [...]
    64 bytes from 8.8.8.8: icmp_seq=578 ttl=53 time=15.5 ms
    ^C
    --- 8.8.8.8 ping statistics ---
    579 packets transmitted, 364 received, 37% packet loss, time 13854ms
    rtt min/avg/max/mdev = 15.009/19.308/50.884/6.152 ms, pipe 2

It's so weird to hear so much complaining about Comcast here. My stats with that same command:

     --- 8.8.8.8 ping statistics ---
     615 packets transmitted, 612 packets received, 0.5% packet loss
     round-trip min/avg/max/stddev = 21.332/35.875/95.576/11.248 ms

I have never seen it be anything different.

> I have never seen it be anything different.

If you parse this statement literally, it has amusing consequences.

Comcast Business connection in Olympia WA:

``` --- 8.8.8.8 ping statistics --- 538 packets transmitted, 530 packets received, 1.5% packet loss round-trip min/avg/max/stddev = 13.706/27.511/101.208/11.207 ms

I'm curious if anyone knows of tools like this that are more deterministic?

I'm in the process of writing a library at work that is basically implementing the CoAP protocol from the ground up and having something that I can script to force the same packets to be lost or delayed is something that I would find very useful for testing. I've been using Apple's network link conditioner, but when I do discover a problem, it can take a long time for the same scenario to happen again which makes testing the fix quite difficult.

I'm currently trying to think through how I'd write this to make the filters easy to setup [0], but if anyone knows of something that already exists, please let me know.

[0] https://github.com/Azdle/dolos

Was this made in response for more real version of https://theonion.github.io/comcastifyjs/ ?

Google Chrome's device mode has a subset of this functionality where you can make Chrome throttle the connection simulating different conditions (GPRS, DSL, offline, etc.).

Access device mode by clicking the smartphone icon on the top left of the Developer Tools (F12) right next to the Elements tab.

It's very useful. I discovered this the other day when testing some client side scripts. I was able to simulate a 3G mobile connection and determine which parts were causing the intolerable load times.

This is a beautifully-named project.

`--device=eth0 --latency=250 --bandwidth=1000 --packet-loss=0.1`

This could have been named `--i-am-in-india`

Did this with dummynet on FreeBSD around 1999. Made it a bridge though, so everyone on the test network could "enjoy" 28.8kbaud dialup experience! Great fun.

Called it "the molasses network", which is what they should change the name to before Comcast smokes them.

SmartOS has this nice utility - ipdadm which does something similar.

ipdadm - administer the Internet packet disturber

https://smartos.org/man/ipdadm

Very nice.

How's this better/different than Crapify?

https://www.npmjs.com/package/crapify

Id change the name, not because of Comcast coming after you, but I guess it's just not very clever IMO. Good luck on the project though. :)

Prediction: name of this project will be different within 12 months due to trademark infringement.

Nice, and the detailed README is appreciated.

I do have to wonder if the name won't cause you trouble though...

I wonder if this can be used to simulate speed-of-light delay for a Mars mission.

Written in Go for nothing, you might as well hack a cheap shell script.

Who cares? It also could have been written in Perl, csh, C, APL, Pascal, or Fortran, and it would work exactly the same.

Can we stop shitting on cool things just because the technology being used is "trendy", or you're sick of hearing about it?

This is great. I would love mobile support.

The project name is pure gold!

In the words of the great poet Macklemore:

"This is fucking awesome"

I have a feeling that we will see an incoming DMCA soon.

Getting downvoted. The lack of humor in Hacker News is astonishing.

Upvote for the name.

Reply on news.ycombinator.com