The case of the 500-mile email (2002) | Hacker News Reader

The case of the 500-mile email (2002)

ibiblio.org

246 pointslibeclipse56 comments

The 500-mile email story seems to be the Hacker News equivalent to Reddit's SR-71 speed check story [1]. It keeps coming back again and again because everybody loves it.

[1] https://www.reddit.com/r/SR71/comments/2dpmw7/the_sr71_speed... (and many others, too numerous to reference)

Thanks, first I've read that and it certainly put a smile on my face.

One of the first articles I remember reading on HN was about an SR-71 [1], and it seems it's by the same pilot because Walter is his crewmate.

[1] http://sploid.gizmodo.com/5511236/the-thrill-of-flying-the-s...

The SR-71 story never fails to put a smile on my face.

Has anyone made a collection of the most re-posted HN articles?

https://xkcd.com/1053/

This new XKCD comic is quite amusing! I read back through several of them and they're so true to my life experiences!

[throwaway account] A couple of years ago, my company (SEA) was bought by a private equity-based company that was assembling a portfolio of similar companies. The IT group got taken over by a competent yet overworked group in their main office (PHX.)

The remote IT group wanted to move everyone's computer to their windows VMs that were in their colo in PHX, which were managed (and presumably logged) out of a central management panel. From their point of view, this was a great solution. I was trying to get them to put a separate VM bank up locally, because of the speed of light delay (at least 15ms) that would be small but annoying.

I had convinced the remote IT people that this would be a good idea, and also the [PHX] CTO apparently thought it was a good idea, but my local 'location boss,' who used to run our company decided that I was just doing it to increase my power. (which was untrue, because I would be going back to just ENG and not IT after the transition.)

So, my boss cancels it and throws me under the bus to management at our new home office. It takes a couple of weeks, then the terminals that log into the remote hosts get installed. The speed of light is one of the constants of the universe, so the problem I mention immediately happens.

My location boss gets annoyed by the slow performance on his computer and, remembering that I'd said something about slowness, storms into my office and starts demanding that I fix the computers. I try and calm him down and tell him that there's no way to fix this without moving the terminals & vms closer together.

He calls up the remote IT folks and they agree with me. He, at his next trip down to HQ, starts slagging on me to anyone who'd listen. Various people at the new home office warn me about this, but there's really nothing I can do about it.

After some hilarity, I'm transferred to a project that was designed to fail so there can be paperwork justifying laying me off.

So that is another way the speed of light can have real-world consequences.

> And I was fairly certain I hadn't enabled the "FAIL_MAIL_OVER_500_MILES" option.

If this actually was a config option, I would bet anything the default would inexplicably be set to "true".

It's like this weird setting in Ubuntu's network driver config file, "11n_disable." I won't pretend to know much about it but every time I install Ubuntu on a machine, wifi internet speeds are slow until I set the above to "8". To me it seems like literally a "go faster" switch.

Only 8, and not 1?

Because, at least on my system, looking at iwlwifi...

parm: 11n_disable:disable 11n functionality, bitmap: 1: full, 2: disable agg TX, 4: disable agg RX, 8 enable agg TX (uint)

So 8 is only aggregated TX, while 1 would be aggregated TX and RX. (It does, however, also inexplicably default to 0 on my system.)

Oh neat, lemme try that out then.

I'd be curious to learn what it does for you, because my intuitive interpretation would say that 1 would be "disable all", but that doesn't explain why "enable agg TX" would be an improvement unless agg RX is worse than the baseline...which is probably a bug.

e: Oh, apparently this is a pathological re-use of an existing module flag to force something on that they disabled due to instability in some cases, see also:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

So I'm sorry, but my assumption about =1 is not valid, as I assumed that all of the bits (1,2,4,8) were mutating the same thing, not "one bit does something VERY DIFFERENT".

Ok, this got me curious... 11n_disable=8?

A quick google turns up lots of people agreeing with you, suggesting the intel wireless driver seems to have trouble on 802.11n. Fine. But 8?

    modinfo iwlwifi

shows us that 11n_disable is actually a bitmask:

    parm: 11n_disable:disable 11n functionality, bitmap: 1: full, 2: disable agg TX, 4: disable agg RX, 8 enable agg TX (uint)

Ahah, your story is starting to make more sense! Setting the value above 8 would set the 'enable agg TX' bit in this bitmask.

Side note: 11n_disable=8 as a way to enable an advanced wifi settings is not the most user-friendly option in the world. It looks like this was added as a patch to fix a compatibility issue with certain NICs[1]. Be careful setting it higher than 8 -- as that would set other flags in the bitmask, potentially turning off 802.11n entirely!

I've not heard of AMPDU before -- but it seems to be an 802.11n-specific protocol extension that allows you to send multiple MPDU blocks (i.e. frames) to the router at a time, rather than one at a time. This seems to trade off latency for error correction between a client and the router [2][3]. So by setting this flag, you're able to transmit multiple frames at a time, rather than retrying each frame individually until it goes through on a busy channel. This really does seem to make a difference, one report online suggests tx throughput goes from 20Mbit/s to 80Mbit/s when agg TX is enabled[4].

Apparently this was introduced in 2014 in order to patch a firmware issue on certain intel NICs. There's a whole conversation on the linux-wireless list about it[5], which seems to conclude that they couldn't reproduce the issue[6]:

> I have seen reports from many users and many devices - and unfortunately I have never been able to reproduce - so I can't know if it affects all the devices. All I can see is that I saw reports on very old devices (5150) and with more recent devices (2230).

[1]: https://github.com/torvalds/linux/commit/205e2210daa975d92ac... [2]: https://routerguide.net/optimize-ampdu-aggregation-on-or-off... [3]: http://eprints.eemcs.utwente.nl/21008/01/eunice.pdf [4]: https://ubuntuforums.org/printthread.php?t=2205924&pp=75 [5]: https://www.spinics.net/lists/linux-wireless/msg118549.html [6]: https://www.spinics.net/lists/linux-wireless/msg118554.html

One day, I would like to understand many of the things you're talking about, but today I do not and hence a lot of my frustration with Ubuntu :P

I can't find the "11n_disable." option. Are you talking about the "11n_disable" ?

The importance of full stops outside the quotes ;-)

When you end a sentence with a quotation you put the punctuation within the quote.

Not in English you don't.

As I recall, according to "Strunk and White", and the "Chicago Manual of Style", all punctuation is supposed to be included inside the quote characters.

I happen to disagree with them, but they have been considered authorities on the proper way to write in the English language for many decades.

American style (Chicago, APA, and MLA) specifies that a period goes inside the quotations.

British style, on the other hand, specifies that a period goes outside the quotations.

http://blog.apastyle.org/apastyle/2011/08/punctuating-around...

As as Brit I was taught to put it inside, but I treat that rule as a strong hint which may be overridden if other complication is happening there...

Fun story - never heard it before.

BTW, I think information travels over "normal" networking gear (e.g. fiber optics) at about 70% of the speed of light (which is why high-frequency trading tends to move to over-the-air microwave networking to shave a few nanoseconds - see [1]). But I guess the story still works, at the resolution it is told.

[1] https://en.wikipedia.org/wiki/High-frequency_trading

I was a friend of Trey back then and I remember him explaining this and thinking "wow, that's some good debugging!"

You'd be amazed at how many people tell Trey that this story is fake. Ugh. There were plenty of witnesses.

Trey made news recently during the flood of negative United Airlines stories:

http://www.newnownext.com/how-united-airlines-almost-ruined-...

TIL: units is a really useful utility command that I had forgotten even existed.

It does induce a geeky joy but it's ephemeral for the most part as Google is vastly superior in usefulness. I learned about units few years back, used it for a while, and I can't even remember when but I just started Googling the conversions instead of switching to the terminal window.

I mostly use Spotlight (on OS X) to do conversions since it saves me switching to the browser or terminal. It doesn't handle as advanced conversions as Google does but it handles 90% of my common cases (including currencies)

Damn, I didn't know Spotlight could do this. Thanks!

I don't think Google can convert millilightseconds to miles at one go. The units program has support for much more esoteric units than Google does.

It doesn't understand millilightseconds, so you need https://encrypted.google.com/search?q=%283+milliseconds+*+sp... or https://encrypted.google.com/search?q=3+light+seconds+%2F+10...

  You have: 100 millilightseconds
  unknown unit 'millilightseconds'
  You have: ^D

Apparently I need to add more units to achieve this level of usefulness.

Something tells me you're on OS X, and like myself, you'll find you mostly want everything replaced with GNU binaries:

  $ brew install gnu-units
  $ gunits

Also see: https://apple.stackexchange.com/questions/69223/how-to-repla...

Well

  user@server:~$ units -V
  GNU Units version 2.11
  with readline, with utf8, locale en_US
  Units data file is '/usr/share/units/definitions.units'
  Personal units data file is '/home/user/.units'
    (file does not exist)


  Copyright (C) 2014 Free Software Foundation, Inc.
  GNU Units comes with ABSOLUTELY NO WARRANTY.
  You may redistribute copies of GNU Units
  under the terms of the GNU General Public License.

  user@server:~$ units
  Currency exchange rates from www.timegenie.com on 2014-04-02
  2866 units, 109 prefixes, 79 nonlinear units

  You have: 3millilightseconds
  You want: miles
          * 558.84719
          / 0.0017893979
  You have:
  user@server:~$

Works for me.

try "3 ms"

A lengthier discussion:

https://news.ycombinator.com/item?id=9338708

Reading this kind of debugging story is kinda like watching a horror movie.

It's very interesting and fun to read if you know the kind of problems you have to deal with are tame and furry in comparison.

But if I actually had to troubleshoot a problem like this, it would give me nightmares for weeks.

Statisticians make really good bug reports!

Maybe in this case, with n=1, but we'd need a better sampling of impoverished departmental heads to support your thesis.

I'm a simple man, I see the 500 mile email story, I upvote.

I thoroughly enjoyed this story, but wonder if the details have changed a bit over time, perhaps? If light can travel ~558 miles in 3 milliseconds, it seems that mail sent more than 250 miles would have failed (because RTT)?

Also, I'm not sure how big a difference it makes, but signals propagate at less than the "speed of light" (vacuum) when the medium is copper.

I've read this story dozens of times and this is the first time SunOS jumped out at me. Around 2001 or so there was a Solaris release (2.7 maybe?) that had a bug in a particular patch release where sleep() returned instantly. That would have exhibited all sorts of interesting symptoms such as this story.

I had the joy of re-engineering a terrible in-house web server because of that bug.

It had used the system sleep to pause execution between the fake threads in their awful program. Suddenly everything was hitting race problems, instead of about 10%. Not something I really understood why it was going so bad, so I just dumped the inhouse implementation of threads, and took up the Solaris one, which was a joy by comparison.

Sleep() returns early if a signal is delivered to the program, so doesn't your code need to handle this case anyway? Of course, this is super rare, so incorrect code could survive for a long time on a correctly implemented system.

You should always handle your errors, but people trust in exceptions and "what works" often enough that we have a culture that doesn't.

http://geocar.sdf1.org/close.html

this is the point of that terrible 'threading' strategy, actually, if i guess what they are doing correctly.

"And I was fairly certain I hadn't enabled the "FAIL_MAIL_OVER_500_MILES" option."

I laughed.

Already really often times submitted. Last time was (I believe) 24 days ago:

https://news.ycombinator.com/from?site=ibiblio.org

[Edit] Typo

It is a rather popular story, indeed:

https://hn.algolia.com/?query=The%20case%20of%20the%20500-mi...

I love this every time I see it. Newcomers may not have seen it so reposting doesn't seem a problem. And 24 days ago it was totally ignored in the new submission lottery, so again reposting doesn't seem a problem.

It was my first time reading about it; it was fantastic.

If you haven't read it before, there's also "The Story of Mel" which is quite a bit older (1983) but is also a very fun read. http://www.catb.org/jargon/html/story-of-mel.html

Yeah, I'll never not upvote this one.

I actually looked up this story myself yesterday, because it was mentioned by the top comment of this thread: https://news.ycombinator.com/item?id=14672174

I wonder if the current OP posted it for the same reason.

Yep! Hadn't seen it before and bet a few others probably wouldn't have either.

That can be said about any hn story.

Stories like this remind me of the days when the internet was young, and fun, and silly, and best experienced in its green-on-black usenet glory. (418 Error: I'm a teapot, for example.) The internet just doesn't seem fun anymore.

It's not that the Internet has lost its fun--it's just too easy to find it these days. You appreciated everything a little more when you had to get on your bike and ride a mile across campus to the campus computing center just to read that silly USENET post on a 80x24 serial terminal. Now you can read it on your watch or your glasses or yell at your home stereo to read it to you.

I used to wonder what it would be like to time-travel to ancient times and show a gun or a car to a Neanderthal. I think it would be just as crazy to show 2017 me to 1993 me.

It was me who approved the merge request replacing our 418 with a boring 400 a few months ago; I'm sorry.

(On the other hand, there's still a server out there returning a redirect 303 Sea Otter, so I guess it balances out)

Reply on news.ycombinator.com