Hacker News Reader

Comment by jordigh | Hacker News Reader

Let me see if I'm the first one to link to that classic story in the same series, "I cannot send email further than 500 miles"

http://www.ibiblio.org/harris/500milemail.html

Or the Magic/More Magic switch

http://www.catb.org/jargon/html/magic-story.html

It's fun when physical reality meets the abstract models that we have built in our heads of these machines.

View on HN

Oh man, this is one of my favorite lines of all time:

> "Anyway, I asked one of the geostatisticians to look into it--"

> "Geostatisticians..."

> "--yes, and she's produced a map showing the radius within which we can send email to be slightly more than 500 miles. There are a number of destinations within that radius that we can't reach, either, or reach sporadically, but we can never email farther than this radius."

I adore when experts use their expertise to analyze real-world things like this and provide ridiculously thorough explanations :-D

it's a useful analysis. Nobody thought of router hops, but this pattern is pretty much what you'd expect, so it was a very good hint.

My favorite story kinda of this nature, of an expert as alien intelligence, was Feynmann's calculations about computer architecture of the Connection Machine:

https://longnow.org/essays/richard-feynman-connection-machin...

It's a few paragraphs, maybe too much to quote, but the bulk of it starts with:

> By the end of that summer of 1983, Richard had completed his analysis of the behavior of the router, and much to our surprise and amusement, he presented his answer in the form of a set of partial differential equations. To a physicist this may seem natural, but to a computer designer, treating a set of boolean circuits as a continuous, differentiable system is a bit strange. [...] Our discrete analysis said we needed seven buffers per chip; Feynman's equations suggested that we only needed five. We decided to play it safe and ignore Feynman.

Guess who was right.

The whole essay is worth reading, if you haven't yet.

I hadn't seen that before, so thanks for posting it. What a great story!

That was a great read. It was also interesting to read about what happened to the company.

https://en.m.wikipedia.org/wiki/Thinking_Machines_Corporatio...

One thought I had while reading this was what areas of technologies are still open to amateurs.

My recent version: I was playing a pinball game in an arcade. One particular ramp shot was registering earlier in the day and then stopped working.

Eventually I realized that the sensor is an optical beam, and the receiver happened to be in direct sunlight coming in through a window! So it was continuously receiving infrared and would never report the beam being blocked by a pinball. Sure enough, it started working again once the sun angle changed by a few more degrees.

You missed an opportunity to cheat the machine by waving your hand between it and the sun. ;)

Heh, but not exactly. If I blocked the sun, the receiver still would have been picking up the real beam. I would have had to block the sun _and_ make the shot with a pinball at the same instant... which is just playing the game normally with extra steps.

I have an optical smoke detector that will give (very loud) false alarms if a sun beam can bounce off a windowsill onto it. It works great if the curtain is closed. Debugging that took a few early sunrises.

Or the "Car allergic to vanilla ice cream" story [1].

[1] https://news.ycombinator.com/item?id=37584399

The 500 mile email story is one of my favorite reminders that, fundamentally, we're still governed by the laws of physics. It's funny, but it's also a reminder that, while networks might be very fast, the latency is still going to be governed by the speed of light.

Well laws of physics is what gave us radio in the first place.

Some of my favorite video documentaries are on how it was theorized and then slowly developed over years and decades until they finally got to spark-gap transmitters.

https://en.wikipedia.org/wiki/Timeline_of_radio

But just imagine listening to spark-gap morse code radio broadcasts for years as amateur and then suddenly someone does a broadcast test of actual voice (violin!) That must have been incredible to hear wirelessly.

24 December 1906 Reginald Fessenden, that was the leap that eventually gave us wifi

https://en.wikipedia.org/wiki/Reginald_Fessenden

If we are doing classic stories - Grace Hooper and the Nanosecond of wire

https://www.youtube.com/watch?v=9eyFDBPk4Yw

https://americanhistory.si.edu/collections/nmah_692464

I actually like thinking about the exchange of physical information as a network propagation delay, and entanglement/coherence as a distributed consensus algorithm. They're kinda samey from a conceptual point of view (in my amateur opinion)

I forgot all about the 500 miles story. My favorite line:

> If the problem had had to do with the geography of the human recipient and not his mail server, I think I would have broken down in tears.

I had a customer who used a line of sight system for extending their network across part of a city.

I had a shortcut on my desktop with the weather for that town ready when they would inevitably call and blame our unrelated equipment for some problem.

I worked at a small, local ISP in the 90:ies that had a point to point link across the river, handling the dial up traffic from the telecom company we partnered with.

Every few days, always at roughly the same time, all incoming dial up traffic would drop. A minute later, the customers could reconnect.

It took a while before we realized that one of the huge passenger ferries that docked a short distance upstream was the cause. When it arrived and departed, its chimneys and possibly bridge and highest deck blocked LOS across the river.

I used to work in high-frequency trading. I had several tabs permanently open to the live weather radar feed for regions where we had microwave towers: the NE USA, the South of England, the Alps...

I'm curious to know where your towers were. Do you know if they still exist? Were your microwave antennae co-located on other operators' towers (e.g. those for VHF radio), or did your company have towers all to itself?

Without going into anything confidential - we had some of our own hardware, but generally rented capacity from firms like [0]. Some towers were custom built for HFT, some were shared with other types of users.

A famous blog post investigating some of the towers as an outsider, at [1], will be of interest to you.

If you want to guess where they are, get a globe, find the datacentres where electronic exchanges operate (it's not a secret: Chicago, New Jersey, London, Tokyo, Frankfurt, Zurich...) and draw the straightest possible lines between pairs of them. Microwaves don't cross the ocean.

[0] https://www.mckay-brothers.com/

[1] https://sniperinmahwah.wordpress.com/2014/09/25/hft-in-my-ba...

Is the microwave setup quicker than going through fiber nowadays? I only mean in terms of latency.

Traditional optical fiber has glass in the middle. The speed of light in glass is only about 66% of the speed in air, so microwave is always faster if you can get a reasonably straight path for both.

There now exists hollow-core fiber, where the light travels down an air gap in the middle, which is theoretically competitive with microwaves/lasers/etc. How much this is being used is a secret, but microwave transmission definitely hasn't gone away.

I used to be in an adjacent field and we used to joke about when the HFT guys were gonna get working on some neutrino detectors & sources to signal straight through the Earth. You could use them for science on the weekends!

Sounds awesome, to be able to send a signal directly through the whole earth from point to point. It also sounds like origin story of X-Men or a new cancer

Reminds me of Hubble and the NRO satellites and how Hubble was an extra that they didn’t need.

Honestly I'm half-surprised no-one has tried this yet.

There was a site with stories like these somewhere, I sadly can't remember the URL any more.

I think the one that stuck out to me was the Soviet mainframe computer that would get weird bit flips almost every day, always at the exact same time. Somebody compared what was different about the days it didn't get bit flips on, it turns out those were the days on which a particular train didn't run, the computer was very close to a railway station. What train was it, you ask? The one transporting the (definitely perfectly safe to eat, definitely not filled to the brim with nuclear radiation) cow meat from Chernobyl. The radiation was intense enough to cause bit flips, I'm sure the quality of soviet components didn't help here either.

Perhaps thedailywtf.com?

What was the one for Apple stories...?

https://www.folklore.org/0-index.html

I think is the one you are looking for:

https://beza1e1.tuxen.de/lore/index.html

Also posted on HN awhile back:

https://news.ycombinator.com/item?id=23005140

Edit: yep, here are your Crash Cows: https://beza1e1.tuxen.de/lore/crash_cows.html

As someone with very limited electrical experience, the more magic switch story instantly went "the second terminal of the switch is probably grounded to the switch casing" when they explained it only had one connected terminal.

This is a very common thing in older automotive electronics, for example.

First thing I thought of too. Anyone know if there a list of more articles similar to these three?

DNS responses sent over UDP are often truncated if the response is too large. This manifests itself as "machine unreachable if name > x characters" sort of errors when you have really long FQDNs.

My wife has complained that open office will never print on Tuesdays!?

https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161...

I remember one (might have been a hn-er's comment, dunno) about the computer restarting when the toilet was flushed. Turns out it was due to voltage drop when a compressor turned on to refill the reservoir of the toilet.

It was:

https://news.ycombinator.com/item?id=39898272

That's why in rural locales with spotty power it pays to have a UPS on any electronics -- you might not benefit much from 15-30 minutes of extra power in a day long blackout, but it keeps everything happy when the voltage fluctuates.

My personal example: VoIP phones stopping after the Asterisk server was up for 3 days.

Reason: the server had IPv6 turned on, and it steadily accumulated privacy IPv6 addresses. These addresses were all sent in a packet describing the supported media endpoints, using UDP.

And yep, eventually it overflowed the MTU and the phones couldn't handle the fragment reassembly.

Which distro was that? ... asking for a friend ...

Just regular Ubuntu. It was around 2009 or so.

Thanks. I've never run Asterisk on Ubuntu. FreePBX is CentOS based which is mostly what I've run Asterisk on. I only started to worry about Ubuntu around 2012.

That mad IPv6 address thing must have stuffed up more than just a VoIP negotiation packet. DNS switches from UDP to TCP when responses get too large.

DNS is affected, but differently. It's mostly DNSSEC signatures that cause trouble nowadays.

SIP is special because the signalling and media protocols are separate. So when a call is being established, the parties exchange their media endpoint locations. This necessarily means that the server has to list its IPs (or DNS names) so that the client can choose the best one. And as a quirk of SIP, it sends the entire set for each of its supported codecs.

"SIP is special".

Yes, I know, just like ftp 8) I do hope that whomever invented putting the control channel in a separate stream from the data is mildly discomforted. Mind you, all that stuff was invented a very long time ago, when trousers were a major trip hazard.

My go to fix is "symmetric RTP", which seems to have become a default over the last decade or two.

I have one first-hand story:

I did tech support via phone for a popular consumer computer brand. One particular call, a woman reported that her computer was restarting every time someone in the house flushed the toilet.

Long story short, her home was in the back-back woods with the home powered by a generator. In addition to powering the computer, the generator was also the source of power for a water pump which would kick on to refill the toilet bowl whenever it emptied. And wouldn't you know that that water pump had a beefy coil around its motor and would brownout the entire house every time it started?

I have a similar one, with an automated monorail hoist. The engineer who started the job had ordered the monorail hoist with a control cabinet with Ethernet comms to tell it where to move (instead of just controlling the hoist directly from the main control cabinet.) After days' worth of shenanigans trying to troubleshoot seemingly random comms drop-outs I'd narrowed it down to only occurring when the hoist was being lowered under load, which led me to the Ethernet cable in the hoist cabinet which ran parallel to the motor cables from the hoist's 6kW VSD. Whenever it lowered, the EMI was enough to nuke the Ethernet connection. Re-routed the Ethernet cable and after that it ran fine.

Here's one attempt I've seen in other HN comments at a shared "awesome list" of these sorts of stories:

https://github.com/danluu/debugging-stories

A user was having a really bizarre problem: They could log in when they were sitting down in a seat in front of the keyboard, but when they were standing in front of the keyboard, their password didn't work! The problem happened every time, so they called for support, who finally figured it out after watching them demonstrate the problem many times:

It turned out that some joker had rearranged the numbers keys on the keyboard, so they were ordered "0123456789" instead of "1234567890". And the user's password had a digit in it. When the user was sitting down comfortably in front of the keyboard, they looked at the screen while they touch-typed their password, and were able to log in. But when they were standing in front of the computer, they looked at the keyboard and pressed the numbers they saw, which were wrong!

The Daily WTF is full of them.

https://thedailywtf.com

https://500mile.email although I do wish it had more content!

Thanks for mentioning! I would love more submissions! I have a few stories in my backlog to read and vet, but not enough. I'll be going through this thread and adding more that haven't been added yet.

The podcast that kills the car stereo episode of Reply All is pretty funny https://gimletmedia.com/amp/shows/reply-all/brh8jm

There's the car/ice-cream/vapor-lock story. Oops, I ruined it.

https://www.snopes.com/fact-check/cone-of-silence/

This is a true classic that never gets old!

Discussed many times here in HN:

1. https://news.ycombinator.com/item?id=37576633

2. https://news.ycombinator.com/item?id=23775404

3. https://news.ycombinator.com/item?id=9338708

Magic/more magic legend lives! I tell that to everyone experiencing a spooky troubleshooting!

i love those stories plus their detailed explanations because you can learn so much from them about technology, physics and even psychology.

and previously discussed: https://news.ycombinator.com/item?id=9338708

I came here to see if someone posted that.

Boy did that [0] send me down a long rabbit hole

[0] magic-story

Reply on news.ycombinator.com