Hacker News Reader

Comment by dumbfounder | Hacker News Reader

This is a secondhand anecdote, but it’s pretty funny. Back in the days of server rooms, a friend’s server for his company would reboot every day around 5pm. They checked everything they possibly could with the OS, they would be logged in and running checks on it and it would spontaneously go offline for about 5 minutes and reboot every day. Finally they decided to go stand in the presence of the server around the time it goes down every day. They watched a cleaner come into the room, unplug the server rack, plug in their vacuum and vacuum around the servers, and then plug the server rack back in.

View on HN

Oh I have a similar one. This one is first hand, I was in the room when we were debuging the issue.

We were developing a smart camera product which were counting traffic on a road. So for example a city council would install this camera somewhere on a road and it would generate statistics of how many lorries, and passenger vehicles, and motorbikes used that road.

One of our cameras exhibited a problem where it restarted every day around roughly the same time. It wasn't exactly the same second though, in fact there was a clear pattern to it. One day it would restart at 19:12:10 and the next day two second later, then again the third day two more seconds later. (not the real timestamp and i don't remember the real time deltas either, but there was a clear progression)

After much debuging we learned that the issue was that as the sun was settling some street furniture projected a shadow in front of our camera. Our software wrongly concluded that it is a vehicle and started collecting information about it for classification. But of course shadows creep a lot slower than real vehicles so it run out of memory before the "shaddow vehicle" has passed out of the frame. And once we run out of memory the system froze and then got restarted by a watchdog.

Turns out the pattern we have seen in the timestamps was caused by the angle of the sun changing which made the shadow trick our algorithm just a little bit later every day.

Definitely an urban legend at this point. A reddit thread[0] mentions it being collected in 1990 from computer stories going around in the 80s[1]

[0] https://old.reddit.com/r/talesfromtechsupport/comments/5yrs1...

[1] https://www.goodreads.com/book/show/3227607-the-devouring-fu...

The story probably endures because things like this actually happen [0, 1]

0. https://www.cnn.com/2023/06/27/us/janitor-alarm-freezer-rens...

1. http://www.st-v-sw.net/Obsidian/Martin/gravity.htm (search sprinkler)

Essentially all these stories are apocryphal. Even this vapor lock story.

Sorry but even before today, having a automotive engineer sent to a random person's home over what clearly sounds like a quack letter seems implausible to me. The dictates of capitalism, human resources, and the politics of the workplace would make this difficult if not impossible. Even in the past when there was more human capital in support positions and more of a sense of customer service.

Way, way too many suspicious stories involve high-level people being involved in trivial issues. I just find it all pretty suspicious. Real stories tend to start with poor customer service at the dealership and being mocked by managers and mechanics. Not some unrealistic ideal white knight manager sending off engineers to people's homes. Imagine how many weird letters a place like pontiac gets. They don't have the manpower to do this if they actually chose to do it, and engineers might balk at the idea of doing at-home support too.

Pretty much any "idealized Americana" business story should set off BS alarms in us. "Oh a trivial problem with your car? No problem ma'am, I'm sending our top engineers over tomorrow," doesn't happen because its costly and unsustainable. Instead ask anyone who has odd car problems. Its endless painful calls and visits to dealerships and mechanics. There's a reason we have lemon laws for cars. Its because whats described in this story doesn't actually happen and people demand restitution.

I don't doubt that someone had a famous vapor lock shopping story (ive heard different versions of this story, usually about a housewife picking up her child from a nearby elementary school), but over the years these stories get modified into memetic structures based on dishonesty because most people are social capital seeking and having a humorous story provides them the immature ego boost they need. So "wow my car had vapor lock when I make quick trips" became "So the CEO of Ford came to my house to look at my ice cream car..." The latter is just more interesting in the market of storytelling.

That is to say, the ONLY reason this story is here is because its been modified to be memeticly attractive. A "boring" (i find old technology faults interesting, personally), but a "boring" story about vapor lock wouldn't make it to places like HN or reddit, which are memetic responders (upvote/downvote mechanisms) and lowest-common denominator (by this demographic) popularity machines. But dress up that boring story and now everyone is repeating it, often times claiming its their story and they know the people in it! The same way the comment you're responding to probably doesn't actually know the famous "unplug the server at 5pm" person.

Wow, you used "memetic" three times. Very erudite.

Frankly I found the whole thing to be shallow and pedantic.

I've swung from -2 to +2 during the day.

You're right -- it was obviously a fun engineering legend, and leave it at that.

Without vast knowledge, many unreasonable things are seen as reasonable. How much of our history's truth is based on what was told to the gullible majority? Should we not talk about Mythology because a skeptic questions it's authenticity?

Comedians make up stories all the time to entertain audiences. These stories don't require accuracy, they are more about delivering specific results; a laugh, a story to share, confirmation bias, etc.

Many people lie, believing they're telling the truth. I think you will have a hard time truth policing people who don't and won't care, but focusing on truth and validity is probably a useful skill for you in many parts of your life.

If I tell you something I believe to be true, but actually isn't, I'm not lying, just wrong. Lying involves an intent to decieve, so if I don't know the truth, it's not a lie.

That is true, in addition, outcomes are not defined by intentions. It was my hope to draw attention to why we perpetuate this behavior (results).

There is a lot of gradation, the differences between stating misinformation you believe is true vs stating what the majority believes to be true due to laziness vs an intentional lie, etc.

There is a word for telling something as though it were factual but with a negligent disregard for its truth: bullshit.

Your lack of faith is disturbing, specially when there are first hand accounts of [Apple/Microsoft/Insert Co] sending engineers to people's houses to diagnose unique problems. This could be a perfectly real story, a different thing would be that you choose not to believe it.

during the PPC 603 era of Macintosh Performa series, we had a bad motherboard, and apple sent an engineer to our house to replace it. This is around the time that the story of apple sending engineers to houses to remove motherboards, raise them 2 feet above a flat surface, and drop them was being passed around. Something about reseating chips that one of their pick and place machines was misaligned on or something.

When I worked my first job in retail in the mid eighties - we were selling Atari ST and there was an official Atari bulletin to do exactly that. It would reseat the socketed chips.

interesting; i doubt my memory is that faulty, so it's possible the story went through some iterations, making it an apocryphal story about apple, instead. Or, possibly, the major pick and place manufacturer had a slew of alignment issues in the 80s!

I thought the apple "lift and drop the computer" story was about the Apple 3. It had no vents, so it was always overheating and "unseating chips" like the Xbox 360 heat issues.

https://en.wikipedia.org/wiki/Apple_III#Design_flaws

And so it doesn't become apocryphal but stays a real story, there 100% were people who "Fixed" their RRoD Xbox 360s by wrapping them in towels for an hour so they cooked themselves even more, or they put the mobo in the oven on a low temperature. That fix was also 100% used for fixing certain graphics cards unseating in the mid 2000s as well.

I've got an HP Laser Printer from about 15 years or so. About once a year, the printer stops responding and I have to remove the board, put it in the oven for 10 minutes or so to revive it.

What even is the theory on how that could possibly work? I believe you, it's just that I don't have any mental model of how baking a circuit board makes it work right for a year, and how it keeps working to fix the issue.

The theory is that 'baking" it could reflow a bad solder joint and fix it. A popular fix of last resort.

I get that that could work and might solve the problem entirely. What I don't get is why you'd ever have to do that every few months.

Laser printers have parts that can get very hot. If their "baking" reflowed the parts, but just barely, maybe those hot parts would just cause the exact same problem all over again.

Ah! There we go! That was what I was missing. Thanks. It all makes sense now.

I've reflowed a GPU in an oven before, personally. with 5 little balls of aluminum foil as standoffs and monitoring the temp closely. It fixed it. I probably still have that - working - card somewhere.

Those might be field engineers, but I have heard of the WiFi teams at computer companies going out to people's houses to test especially weird situations.

It's a fun story to read, whether it's fact or fiction. We are not too concerned about whether a large company would really send an engineer to investigate a seemly absurd claim, in the same way that we are not too concerned about whether one single shoe is able to uniquely identify Cinderella and no one else.

Every now and then a ceo will do somethings like this. If this is true idk. Good for PR.

I've had the CEO of a municipal water company at my house looking at what his contractors did to my property frontage and potentially my well. Also, my other story in this thread is documented in posterity on a forum, you can see i didn't embellish any of it, and that involved a manager or supervisor having to drive quite a long time to my house at 10 PM with a technician to source a problem i had had for weeks or months.

So, in essence, "it depends". Good stories, in the memetic sense, will have hooks to ensure that the moral or point of the story is remembered; in a great story, the memetic hooks are so great that you can repeat the story nearly verbatim to other people, after hearing it yourself.

> Essentially all these stories are apocryphal. Even this vapor lock story.

The vapor lock ice cream story might go back to this 1997 Car Talk Puzzler, where Ray claims it was a customer of his: https://cartalk.com/radio/puzzler/finicky-volare

I used to work for General Motors as a field engineer. Basically, I was the "mechanic of last resort" for some issues. The most certain method of getting something fixed was to write a letter to someone at the top, or very close to the top. When the request rolled downhill to "fix this", no one knew if this was a whim or something serious (like it was the CEO's neighbor). So those service requests got absolute priority.

I remember one incident where that "fix it" letter came from high enough that I drove out to the customer's house and swapped out their radio in their driveway. At night.

When GM bought EDS, Ross Perot ended up on the board. He'd do all sorts of silly things. Like when something went wrong with his car, he'd take it to a dealership. And report back to the board how that went. The first few times, they just told him to hand the keys to the valet at the executive garage and tell them what needs fixing. The plant I worked out of made radios. Other branches of the division that I worked for made engine computers and instrument clusters. If someone at the executive garage had a radio problem, one of my tasks was to go out to the assembly line, grab a radio, test it, then get it to the Kokomo airport for the GM jet to pick up. FedEx (tagline: when it positively has to be there the next day) wasn't fast enough. That fleet of jets would carry parts from plant to plant. And the executive garage was the best equipped and best staffed GM dealership on Earth.

You soon learn at a big company that almost any expense is justifiable to the boss if it prevents his boss from asking inconvenient questions.

By coincidence, I have witnessed this just last night. No details will be provided to protect identity, I'll just say you probably heard of the company.

It's staggering.

What industry?

manufacturing.

You don’t happen to be the guy who overnighted a bunch of network cables for $5,000 because the boss had to have them overnighted and the company didn’t care what it cost I read about recently? lol

Oh for these silly escapades to only cost $5k.

Delco! I now live in one of those small factory towns in Indiana. It’s fascinating to me how so many little towns in the Midwest existed just to make one small part for Detroit.

Well, CEOs do look at these letters quite frequently. At least, I did when I was the CEO of a networking company.

We were only selling equipment in the US at the time but some had found its way around the globe. This one particular Indian gentleman had been engaged for some time with support claiming that his unit wasn't working right and exhibiting all kinds of strange behavior.

I had a habit of looking at the cases with the longest open history, which is how I found it. I continued to monitor though I didn't send anyone out to India or anything like that.

Eventually it came out that the sleek looking aluminum unit had some kind of stocky packaging material stuck to it, so the customer had put it in the washer to clean it off.

>>all these stories are apocryphal

Do not mistake cynicism for intelligence. It is actually negatively correlated [0, + other studies]

Just listing a basket of "I doubt..." and "No one does..." does not make it all false.

Of course Just So stories exist, but that does not mean that all cool stories are fabricated Just So stories. In this case, it is the cynicism that merits a [Citation Needed] tag.

[0] https://www.psychologytoday.com/us/blog/trust-games/202111/t...

It's not just any memetic structure though. Urban legends have a number of components to them that make people want to re-tell them. See, for instance, my other comment here: https://news.ycombinator.com/item?id=37596325

One of the things I talk about is that these stories often ride the edge of plausibility. Even with this one, dispatching an engineer to someone's house over a "quack letter" seems like a thing that could happen or could have happened in the past. Having high level people involved is a thing that others have talked about, but I'd also like to point out that people have literally emailed jeff@amazon.com before about issues and gotten resolutions that way, so it's not completely off the mark.

The real clue here that we're dealing with an urban legend and not an actual incident (though it may be based on one) is the lack of specifics. Urban legends are often not really pinned down to any specific time or place.

would it be surprising if this has happened more than once?

I mean it actually happens. Not powering down the actual machine, but in my last house the landlord had a cleaner visit every fortnight. Occasionally I came home to find downloads broken or my remote connection would drop randomly in the afternoon. We lived in an old house and had a wifi extender to reach the upper floors (and not enough sockets). The cleaner would unplug the extender to vacuum the kitchen and half the house would go offline.

The ice cream story reads too much like a dramatisation to be truly believable, but accidental and repeated unplugging is common I expect.

The version that I remember hearing was https://www.snopes.com/fact-check/polished-off/. Also false.

I've never heard that one. It has many more characteristics of a modern urban legend than the simple "janitor unplugs the server then plugs it back in" story. Urban legends are essentially modern day folk tales. They'll often involve fear, horror, or even humor, and they're frequently cast as cautionary tales. They tend to be reasonably detailed (which helps keep a listener's attention), but at the same time they frequently not pinned down to a specific time or place. They're almost always told as secondhand stories (e.g. it happened to "a friend of a friend"), and tend to fall at the edge of plausibility.

But, the key thing is that urban legends always reflect some sort of underlying societal belief or worry. "Janitor polishes off patients" is about the fear of death, and how it can come for anyone at any time, and there's nothing they can do about it. "Janitor unplugs server every night at 5" is probably more about the idea that strange things happen sometimes, but it usually turns out that there's a good explanation.

I think the vacuum story gets retold because it's so close to real-life stories.

The UPS on a server I manage would trip once a week around the same time. The old story came to mind and sure enough, once a week it was time to vacuum and someone would plug in a vacuum cleaner into the same circuit (700W vacuum cleaner on a 100V/15A circuit), causing enough voltage dip to kick the UPS into gear.

I personally had a similar situation. We had an accounting firm whose servers we were maintaining, and occasionally the print server would reboot, always between 9 and 10 am. So I sat in there for a week between those hours and noticed the light would dim and occasionally the reboot would happen.

It turned out that the lawyer next door would come in, turn on his PC, printer and coffee pot simultaneously because they were all on the same power strip, and the drain was causing an undervoltage on the circuit the server was on during startup. We had it on a UPS, but it turns out that at the time consumer grade UPS systems only handled outages.

I measured drops as low as to 85 volts, in practice anything under 95 or sou would reboot.

1999, the town hall of a small municipality s/e of Stuttgart Germany. For the 3rd time in half a year me (IT apprentice) and senior developer are onsite to fix one MS SQL db's broken indizes. It's the early afternoon of friday and humid hot, a thunderstorm is on it's way. We toil on and are almost done when at about 16:00 the chief officer turns up and informs us that we have to hurry because at 17:00 all electricity will be cut in the building! Senior developer and me share a shocked glance. "Is this every friday?" "Why yes! We conserve lots of energy that way." Senior and me share another glance and on impulse I speak up. "Isn't the weekly db cleanup scheduled for 16:30 PM on fridays?" The senior developer nods. We spend the remaining hour to fix the problem and then cancel this weeks schedule and manually shut down the DB server for the time being. Every friday these guys cut the power to the building, causing all kinds of production issues with not only their Netware Server but also the DB server for their collections software (our product). As per maintenance contract we were required to repair the DB everytime which was a hassle and a waste of ressources on our behalf. In the end it took another month or so until they relented and rescheduled the hard power down until saturday morning so that all servers could shut down properly even AFTER the backup jobs could complete (backup was also totally broken but they never did check or even perform a restore so they never actually noticed).

Man I've heard this story so many times, this guy must have a million friends in a hundred countries he's told this story to. :-)

funny thing, something like this happened to me. we were on site doing some implementation and went back to the hotel at about 3am.

less than an hour later, we get an email from nagios (i think it was nagios? it was a GOOD while back) complaining the server was offline. we got into a cab and went back straight up (this server was not supposed to be offline ever).

guess what? a maintenance guy turned the server off by mistake while cleaning up the server room -- even worse, he was not even supposed to be there!

this triggered a bunch of security checks and the company found out that most employees had access to any room in the building.

The key to making friends is having a good story.

I think it's more like a common experience. My wife and I both work, so we have cleaners come in every two weeks for a deep clean. We've used several different companies, and apparently it's standard practice to unplug things when a power outlet is needed. They don't unplug computers that have monitors, and they don't unplug things with visible clocks that would need to be reset, so they do take some care not to inconvenience us, but they'll unplug anything else, including NAS appliances, DVRs in the middle of recording shows, etc. When we hire a new company, we make sure they mark down a special request that they not unplug anything, pointing out that we have ample outlets and can help them find one or free one up if necessary. I also replace any network connectors that lose their little plastic locking tabs, because they're likely to slip loose when things get jostled around during cleaning.

It is a common story and sometimes those get put in the collective blender and we get apocryphal stories out of it. Here's two stories of my own:

Back in the mid 90s, I built out a system that gave every school in a district their own webpage that was carved out of some government funding for providing internet access. There was no budget for hardware though, so it ended up running on a repurposed workstation in somebody's office. One Tuesday even the cleaners unplugged it to vacuum and it didn't power back up after being plugged in. On Wednesday somebody helpfully stuck a piece of paper saying "don't unplug" to it, which seemed to solve that problem until the whole project was mothballed.

In the late 90s, I worked at a company where we started getting complaints from the staff about machines being getting slower over time. Nobody took it seriously until there was an inventory of machines taken and we found that a large amount had significantly less memory installed than they should have, somebody was stealing half the memory sticks from each. Hidden cameras were installed in the office and it turned out that somebody on the cleaning crew came with a screwdriver and ESD bags and knew how much to take to leave the machines working.

Goes again to show the usual thing that gets you caught is repetition.

Had he struck once or twice and then left the rest alone nobody may ever have figured it out.

I would expect the repetition generally comes out of necessity. If he's selling the parts to feed himself or his family he's that much less likely to choose to stop if it means giving up some source of income, however ill-gotten.

It's more likely from "I got away with it, I'll get away with it again" - but I've not done deep research into thefts, but the ones I know were along the lines of "I need more money for more drugs, this will get me some money".

For critical devices you could use red outlets. Different colored outlets are scary.

Similarly, we were just finishing up a project for a client and we’re just going through and connecting everything up.

The hardware consisted of 6 color terminals (Wyse 350s I think), the server, and 10 PCs with large, 20” tube monitors. We had these all in my bosses office powered through assorted outlets and power strips. The room was wall to wall machinery.

It was after hours when the janitor came in and plugged in his vacuum. I looked at him, he looked at me, I looked at my boss, who looked back. We then both turned back to the janitor, who looked at us.

Not a word was said as he fired it up. 2 seconds later, the power was out, we blew the breaker. Took us a half hour to get it back on. But it was funny at the time.

This story is my favorite in the thread. Thanks.

> Back in the days of server rooms

Huh? Yesterday? Or are you referring to the fact that servers are now often offsite, in 'clouds'?

Pretty much, server rooms still exist in offices but they'll mostly be for the network infrastructure, with most workloads including email and storage having moved to the cloud. Office365 is a really tempting offering, compared to operating your own servers + staff.

I had a similar experience: I have a permanent dynamic lighting system installed in a food hall / night club. One day about a year after installing the owner called to tell me a fixture was on the blink. The following day, another, the following day, more, but different ones. This was worrying as all the hardware is custom and would be very hard to source replacements for. We investigated lightning strike or other power surge related to construction on the block, water damage, etc. Long story short we found out that the new cleaner, who was VERY VERY thorough, was pulling out the server rack to mop behind it nightly and straining the cat cable terminations.

This story is older than the internet.

this is why denmark has an additional slanted plug for IT equipment and a winky one for hospitals

In the US the "winky" receptacles that go (-|) or (⊣|) - except the T faces the other way - are 20 amp outlets, 120 volts. every receptacle that differs from the "shocked face" means something. Our standard shocked face receptacles are 15A. In hospitals, you need a guarantee that your monitors or assistive devices aren't going to trip a breaker somewhere if someone plugs in a vacuum; making it impossible to use the receptacle for something else is something that only matters after it's too late, i'd think.

In hospitals, they're colored differently to denote which sockets are on backed up lines vs not.

How can, say, a cleaner tell the prong configuration on a plug when it's plugged in and not visible?

none, but they sure cant plug in their vacuum.

hispital ones are red wires

Right but the point isn’t to stop something being plugged in, it’s to stop something unplugged

Reply on news.ycombinator.com