The 500-mile email (2002)
web.mit.edu
web.mit.edu
It's one of those things that I highly doubt would have occurred to me to have even checked, or given even a moments thought to, under normal circumstances.
Over the years, my Google-fu has failed me. Any clue? :)
For one thing, I'm pretty certain the story involved recovering a deleted file's inode via some other running process's FDs.
Best I can claim is zmodem transfers of uunecoded packages over a PLIP link as I tried to get ethernet support up on an old but fairly reliable box.
[1] http://blogs.technet.com/b/exchange/archive/2004/04/08/10962...
The bit about the recipient processing bug is novel tough, ouch.
http://www.metafilter.com/78177/PLEASE-UNSUBSCRIBE-ME-FROM-T...
I remember it happened at NYU a couple of years ago and they turned it into a kind of ad-hoc social network/partyline. I wonder if anyone archived those emails? I suppose they deserve to remain "private."
"Bedlam may refer to:
Bethlem Royal Hospital, London hospital first to specialise in the mentally ill and origin of the word "bedlam" describing chaos or madness"
How else do you propose handling BCC and mailing lists?
I am actually surprised the sysadmin in this scenario thought it was a bad thing that the statistics department did their research and presented a well documented error.
On the other hand, there's the word document with nothing but a screen shot showing half of a useless error message.
Example (contrived):
$ seq 10 20 | paste -s -d +
10+11+12+13+14+15+16+17+18+19+20
$ seq 10 20 | paste -s -d + | bc
165
$ cut -f 3 -d , something.csv | paste -s -d "|"
a|b|c|d|e|f
$ egrep "$(cut -f 3 -d , something.csv | paste -s -d "|")" another_file
... result lines appear here ... $ seq -s + 10 20
10+11+12+13+14+15+16+17+18+19+20
# a few random samples for an IN SQL statement
$ shuf -i 1-500000 -n 5 | paste -s -d ,
371492,250061,266669,455846,295852
# we can even get PI
$ ( seq -s + -f '4/%g' 1 4 100000 && seq -s - -f '4/%g' 3 4 100000 ) | paste -s -d - | bc -l
3.14157265358979523735 echo '4*a(1)' | bc -l
paste - -
ls /usr/binThough it doesn't contain `units` either, here's a Wikipedia list of the standardized (IEEE 1003.1-2008) unix commands. http://en.wikipedia.org/wiki/List_of_Unix_commands
I felt for the author as I got deeper into the faq, and recognized this pattern of cynicism, then decided the author was so generous and thorough, not out of obligation (make the emails stop!), but because that is the type of detailed person he is -- and good at dinner parties too!
A critical rule, however, is to omit detail, (a reader is unlikely to question an explanation they make up themselves) and most importantly, to omit details you know to be wrong. (It is impossible to nitpick a statement that is never said)
An odd feature of our campus network at the time was that it was 100%
switched. An outgoing packet wouldn't incur a router delay until hitting
the POP and reaching a router on the far side. So time to connect to a
lightly-loaded remote host on a nearby network would actually largely be
governed by the speed of light distance to the destination rather than by
incidental router delays.
It's also a reason why short business emails are better than longer ones. You can always go more in depth. It takes skilled restraint to touch on only the most relevant details without losing the larger point.
But as I wrote in the FAQ, I fired off that email in under an hour in reply to a fast-moving thread on an email list where people knew me by reputation and wouldn't question my skills; the totality of my "research" was trying to reproduce the original numbers from memory ("500 miles" stuck in my head, but the distances to the places I remembered pinging did not); and I didn't ask anyone else to edit it for me.
All of that would have been ridiculously unprofessional of me as a writer for something intended for as wide an audience as it went to. But I had no idea it would be forwarded so much and so often (nor so many years later, now decades after the original event!).
And that's why I really prefer it when people link to the canonical version I maintain at http://www.ibiblio.org/harris/500milemail.html with a link to the FAQ.
Post-mortems are a case where you must have both story and correct details: lack the first, and you won't create change because the people who need to know in order to implement the required recommendations won't read the whole thing (or retain it later); lack the second, and really, how can anyone trust your recommendations?
Here, I was just trying to quickly bang out a funny anecdote. The things that stuck in my mind I could use to reverse-engineer numbers. I did this because—at the time I worked the incident—I was working with real numbers, so the story needed them for verisimilitude, to give a sense of what I was wrestling with. If I'd had any clue this mail would have taken on such a life of its own, I would have been more careful with them and gotten a tech reviewer and copy editor before posting.
This gets posted on some forum or another several times a year; for a long time I had a Google Alert on it and would hop in threads whenever it happened, since it always followed a common pattern:
1. Someone posts a link to the story, but not my canonical copy with a link to the FAQ.
2. More trusting and/or less-technical respondents upvote or forward or Like or +1 or quasisuperplauditize or whatever the medium has until it gets notice from...
3 ... less trusting and/or more-technical types, who expose the "flaws", most of which are covered in the FAQ.
4. Someone thinks to do a Google on "500 mile email", which returns as the top two results my canonical copy and the FAQ, and posts a link.
5. Most people lose interest while a few continue to squabble over ever-finer details.
Depending on at what point I jumped in, I could affect the speed of the above cycle, but it never changed the cycle itself. The fun of the story is following me through my own emotional cycle I felt when I worked the issue, starting with the initial "no way" to "you're having me on, right?" to "maybe...", to "dear God, this is actually happening", to "I must be going crazy", and finally to "Eureka!"
My intervention in the above cycle really wasn't adding that much to the enjoyment of the story, so I stopped doing it. (I'm not sure it's adding anything today, either, but Hacker News is an important enough forum for people I respect and care about that I thought I'd break vow and rejoin the fray this once.)
Also just discovered the "units" conversion program and disappointed that the default Mac library has only 586 units. And shockingly there don't seem to be compatible libraries out there.
Edit: You'll then want to run it with `gunits` rather than `units`
For Mac, try brew install gnu-units and then run gunits.
I suspect that a lot more can convert millilightseconds to miles out-of-the-box now at least in part because of the popularity of this story over the past 13 years.
See https://futureboy.us/frinkdata/units.txt for that.
1) Add this line under the lightyear definition in /usr/share/misc/units.lib (or wherever `man units` says the standard units library is under the FILES section)
lightsecond lightyear / 365.25 / 24 / 60 / 60
sudo mv units macunits sudo ln -s $(which gunits) units
But then it sent him off in a direction not worth going. He literally started to map out how far emails would go if they succeeded. The whole time the error was in the timeout instead.
But I tend to agree with your conclusion. Damn statisticians! :)
> unknown unit 'millilightseconds'
Is this one of the embellishments that just makes the story more entertaining?
Via 'man units': "The conversion information is read from a units data file that is called 'definitions.units' and is usually located in the '/usr/share/units' directory."
Via definitions.units (L. 223), you can see the milli- prefix: https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-...
Via deifnitions.units (L. 1060), you can see the lightsecond unit: https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-...
Maybe check it for completeness?
Edit: Spelling
lightsecond lightyear / 365.25 / 24 / 60 / 60
Here's the line you'll want to add:
lightsecond lightyear / 365.25 / 24 / 60 / 60
Q: "Your email server for some reason is only working for addresses within 500 miles of the server. What may go wrong?"
And let the candidate think logically and reach some sane answer, even if not 100% accurate (i.e. check routers first, connectivity, DNS, timeouts...)
If I were the sysadmin and that happened, I would need to have a meeting with some people. What's the point of being a sysadmin if he operating system is randomly going to be completely changed without someone telling you?
I have a fair amount of built up rage. This seems like one of those situations where it is actually your responsibility to rip people a new one.
> brew install gnu-units > gunits
2 years ago: https://news.ycombinator.com/item?id=6296201
3 years ago: https://news.ycombinator.com/item?id=3802458
4 years ago: https://news.ycombinator.com/item?id=3114391
5 years ago: https://news.ycombinator.com/item?id=1293652
6 years ago: https://news.ycombinator.com/item?id=385068
7 years ago: https://news.ycombinator.com/item?id=123489
Bet there are a few more that will find this submission too.
That said, it's wise to consider the frequency with which such things appear, individually and in total. Too much repetition and focus on memes becomes dysfunctionally self-obsessive. Not sure what the right answer is, but I can probably deal with once per year, short time on front page, and small % of total content.
The book also holds, iirc, a passage about a group of chemists sitting down for lunch when something on one of the shelves start getting uppity. They barely had time to dive under the table before the container started bouncing around the room.
It really makes me sad the BOFH series of stories is over, I loved those too.
I missed it all the other times and am glad it was reposted.
Or perhaps senility is setting in early. :-D
Say what? Nobody writes a sendmail.cf from scratch, unless they are crazy.
> ... that used the nice long self-documenting option and variable names available in Sendmail 8 rather than the cryptic punctuation-mark codes that had been used in Sendmail 5
Good system administrators stick to conservative, portable subsets of configuration and scripting languages, rather than bleeding edge stuff.
When they deviate, they have a clear plan. They document their choice to use something new and shiny, and they keep it separated from the default system configuration.
Since SunOS came with Sendmail 5, the upgraded Sendmail 8 should have been installed in some custom location with its own path so that it coexists with the stock Sendmail, and is not perturbed if the OS happens to upgrade that.
A good syadmin would stick that in some /usr/local/bin type local directory, and not overwrite /usr/bin/sendmail.
The consultant was not wrong to update the OS. People have reasons to do that. The consultant should have consulted with the sysadmin, of course. But even in that event, it might not have immediately occurred to the sysadmin what the implication would be to the sendmail setup.
My biggest problem with the author was not that he uses his admin blunders as a basis to call himself a good sysadmin, but that he assumed that the stats people were idiots who don't know anything about `puters or networks.
I was not surprised by the 500 mile claim. It strikes me as obvious that the 500 miles has to do with some combination of network topology and propagation delays, those being approximately the same in every direction.
Yes, networking does work "that way": farther places take more time to reach than nearer ones, broadly speaking. (Of course, it's faster to reach something 12,000 km away with no packet switch in between than something 50 miles away with switching. That doesn't eliminate the generality.)
It was also obvious why they didn't report the problem instantly; you cannot instantly know that mail isn't reaching beyond 500 miles without gathering data and correlating to a map, which takes time. Instantly, you can only know data points like "I can't mail to users@example.com". You know that if a stats person gives you a number, it was based on data, and not just a couple of data points. The head of the stats department isn't going to give you a number that isn't factual and backed by science. Of course stats people pride themselves on their data analysis; they are not just going to relay a couple of data points with no analysis attached.
But I think your criticisms seem a little uninformed (or possibly over-informed by later practice to the point where you aren't considering this in the context of mid-1990's practice). Let's see...
> > And also being a good system administrator, I had written a sendmail.cf [...]
> Say what? Nobody writes a sendmail.cf from scratch, unless they are crazy.
I didn't say "from scratch". I used the m4 macros to create a cf, like everyone did at the time. Using the default file would only work if you still used email programs that read raw mbox files, had no email lists, and needed no interesting aliasing or vacation script behavior. Oh, and ran in an environment where it was reasonable to assume someone's canonical email address could be found via the equivalent of "echo "${USER}@${HOST#.}".
Very few production systems could get away with that; writing a sendmail.cf was standard practice. And with m4, you usually spoke of "writing" a file where today we'd call it "configuring" a file; either way it was taking boilerplate and replacing bits with things that were right for your situation. I assume you wouldn't have had an issue with my writing that I'd "configured" the sendmail.cf. That's all I did.
> > ... that used the nice long self-documenting option and variable names available in Sendmail 8 rather than the cryptic punctuation-mark codes that had been used in Sendmail 5
> Good system administrators stick to conservative, portable subsets of configuration and scripting languages, rather than bleeding edge stuff.
Hmm, you either weren't administering SunOS in the mid-90's or you're forgetting some details. SunOS still came with Sendmail 5 years* after best practice was to use Sendmail 8. Check out the O'Reilly Sendmail book of the time's pagecount: it was longer than the prior and the later versions because it had to document both. I'm not entirely certain SunOS (as opposed to Solaris) ever was upgraded to Sendmail 8 in the distribution; obviously the people using SunOS still so late were change-averse.
"Bleeding edge" != "the version that all but the most conservative holdouts are using". Also, remember that this was the same period we were doing the rsh/rlogin conversion to SSH. Sendmail 5 still had known security issues that were fixed in Sendmail 8. We were used to replacing system components when what the OS vendor was shipping us was literally dangerous to run.
And Sendmail 8's Sendmail 5 compatibility mode was simply there for testing; it was never intended to be used production long-term, so using a least-common-denominator sendmail.cf wouldn't have been "conservative and portable"; it would have been risky, bordering on malpractice.
> Since SunOS came with Sendmail 5, the upgraded Sendmail 8 should have been installed in some custom location with its own path so that it coexists with the stock Sendmail, and is not perturbed if the OS happens to upgrade that. > A good syadmin would stick that in some /usr/local/bin type local directory, and not overwrite /usr/bin/sendmail.
Again, either you didn't run this installation in the mid-90's or you're forgetting some details. /usr/lib/sendmail (notice the "lib"! Your referring to "/usr/bin/sendmail" suggests to me you definitely weren't running SunOS 4 or have forgotten details; sendmail was never in /usr/bin) couldn't be left alone, as other tools hardcoded that path. The actual executable was there, so symlinking couldn't be used to get around that.