Hacker News Reader

Comment by vidanay | Hacker News Reader

We once had a customer who would call us in a panic a couple of times a year saying our inspection equipment was experiencing unusually high false rejects and they were generating very high scrap rate. By the time we got a technician on site the next day, everything was working flawlessly and the customer couldn't reproduce the problem either. This went on for almost three years with various levels of escalation to the current management. Finally, one day a technician was on site for another project when the customer came up to him and said "It's happening right now! Come fix it!" The technician rushed over to the equipment and discovered that the sun was shining at exactly the right angle to cause a lens flare in one of our cameras. This happened twice a year as the sun moved along its trajectory. A strategically placed piece of opaque plastic fixed it permanently.

View on HN

I knew a guy who was a computer tech early in his career. One of his rounds was on a military base, and they had just moved their computer room up one floor. They started having problems with their tape drive, it would just randomly pop up an error while they were using it. He tried mightily to diagnose the problem but couldn't figure it out. Finally he took a break and walked to the nearest window and looked out. He saw a radar antenna making a sweep - and realized the error came when the antenna was pointed in their direction.

I sent my Commodore 64 for repairs more than once only for them to find nothing wrong until we realised it was the buildup of static electricity from being stored under our big TV - it "broke" when we didn't use it for a while but watched lots of TV. It'd be "fixed" by the time it took before the repair people got around to it.

The symptom was that it started "typing" automatically.

That’s one downside of static typing.

boooooo

what do you know about true humour? :P

Well done

It truly is well done to make a cheap joke on HN that hits just well enough not to get downvoted to oblivion.

I had a weird 'vibration' in the monitor for my first PC (this was after I caused it to decolor it by passing a speaker too close by it); took it to the shop, no issue. Back home, issue. I believe it turned out to be a power brick that was too close to the screen or its cables.

Reminds me of the sun outage that affects Indian stock exchanges (BSE and NSE) at certain times of the year. VSATs used by traders experience loss of connectivity to geostationary satellites while they transit the sun.

https://en.wikipedia.org/wiki/Sun_outage

I had a similar issue in my industrial automation class. We were sorting cylinders by diameter and height as they went down the conveyor belt. PLC controlling motors, sensors, etc.

My group got everything setup, built our program, and everything worked fine. Waited a few minutes for the TA to verify, but it failed. We changed a few things, it worked, but failed when he came over.

Another group looked over our code, no issues noticed.

Finally I realized I was standing when we were testing things. I sat down waiting for the TA to verify. My shadow blocked the sun from the photo eye. Wasted half the lab on an issue that was entirely dependent on our position in the room, but found the root cause.

>Finally I realized I was standing when we were testing things.

What clued you in to this possibility?

>Wasted half the lab

I don't think it was entirely wasted time, though. You can't plan to teach that kind of lesson ("Look outside of your usual blinkered problem-solving-space"), it happens when it happens.

As this whole thread shows, most of us learn it during our careers at some point but you were lucky enough to learn it before you even started.

You get the same problem with geostationary satellites, e.g. for satellite TV.

There's period of about 10 days every spring and fall where, for up to 30 minutes every day, the sun transits 'behind' a satellite within the beamwidth of the dish and totally overwhelms the signal at the LNB.

So an eclipse…of sorts

In this case, transit is the right jargon.

A transit is when the foreground object doesn’t completely hide the background object.

An occultation is when the foreground object hides the background object.

These two situations are collectively called occlusions.

An occlusion is an eclipse if the the observer falls in a shadow.

https://en.wikipedia.org/wiki/Occultation

I experienced this when I was working on a cable company but sometimes they give out notices when it's going to happen. Random stuff still happens because of the sun. The sun is a very scary thing if you are studying it everyday but most people don't know it.

Once upon a time I had a problem with remote control. It woukd stop working from different positions, but from time to time, not always. It took a while for me to realize there’s a heating radiator behind my back in that directions. And it went hotter or colder depending on thermostate. I guess at one point its IR output would overwhelm IR output from remote control.

When I first read this comment I was picturing a hot water radiator behind you, so it sounded unlikely based on the blackbody radiation curve for ~320 K.

But now I realize you're probably talking about an electric heater. Given that their heating elements get so hot that they're putting out visible red/orange light, it seems very plausible that an electric heater could produce enough ~940 nm IR to drown out the signal from the remote.

Thanks for sharing! I will try to keep this in mind when I'm troubleshooting IR remotes in the future.

As I try to recall, it was not typicall remote, the problem was between Nokia 6310i and IrDA transmiter, since that's how you connected phone to the PC back then. Looking at wikipedia, that IrDA was small range, probably low powered, so maybe even very low IR source in the background could create problems? IIRC problem was with the water radiator, which surprised me a lot back then. But we did use electric heaters as well at the time and my memory could be wrong.

Btw. that Nokia 6310i from 2002/3 is still used today by mother-in-law, and original battery still holds for 4-5 days!

This is like a real-world equivalent of the "cleaning lady unplugged the machine" urban legend.

I was going to say it was the real world equivalent of, like, Redwall, or one of those other fantasy books where an event happens once a year when the sun shines in exactly the right spot to illuminate some secret writing.

Indiana Jones, Raiders of the Lost Ark.

The hobbit

Witcher

There is the Anthem Veterans Memorial that projects an image on the ground on November 11th at 11:11 and at no time else.

https://en.wikipedia.org/wiki/Anthem_Veterans_Memorial

Except for when it is visible at other times, as that link explains?

Isambard Kingdom Brunel built a rail tunnel in England which has the sun shining straight through it on his birthday

I thought it was an urban legend and then somewhere in 2005/2006 I first spoke to one guy who claimed it was a bank in Tønsberg, Norway.

Later I worked with a IT manager who also confirmed it was such-and-such bank in Tønsberg. I have forgotten the name of the bank but I am still on friendly terms with him so I could ask next time I see him.

(My first draft of this post said that the first bloke had claimed to be in the room, but this is 15 - 16 years ago and the next story is also close to a decade ago so I might have mixed up who said what.)

Maybe she overheard they need to sort out bugs and she was like "bugs? in a computer? not on my watch! grabs hoover".

I knew someone who had it happen at his office either ca. 1995, in Oslo.

It was very much not a proper server room, which made it more understandable that they'd not realise there was stuff there that shouldn't be turned off.

I suspect there's been plenty of real-world incidences of stuff like this, and that many have never been noticed.

I've certainly pulled the wrong cable myself a couple of times over the years, even after thinking I'd been very careful, and being aware that I was dealing with servers that should stay up.

I know someone who had that happen to a server he was directly responsible for (well, someone unplugged it anyway, might well have been someone else in the office - it wasn't exactly a proper server room)

It was not just that the server crashed, but the ventilation proved to be bad enough that one of the SCSI drives (which should have been in a RAID, but wasn't...) wouldn't start.

They ended up opening it to try to kickstart it manually (been there, done that myself; had a drive survive 6 months with me "helping" the motor spin it up every morning; yes I backed everything up very regularly during that period), finding the drive head had gotten stuck to whatever material covered the plate. They ended up putting the drive in an oven while connected, and heat it until it spun up, and which point they dumped what data they could.

I do have a memory of having to bump an old heavy drive to get it to start spinning. :-)

I opened mine up, so I'd start rotating the platter by using my finger to start spinning the centre.

Of course this was with a 20MB hard drive - the sensitivity to everything from dust to alignment changes was magnitudes different from the internals of modern drives...

It is reported in the book Absolute Zero Gravity, about having happened in a military context, while others here mention a bank: the only reason to believe it may have happened so infrequently that some call it a urban legend is that in the past that infrastructure was rare. The rest - failures in the workflow of instructing and monitoring "innocent" personnel and contractors - is an "overly" normal factor.

I had a root cause once that was a cable draped across the corner of a ventilation duct.

The duct would vibrate when the air was on, and the corner was pretty sharp, which caused the duct corner to 'saw' its way through the cable's insulation over time.

Took a while to isolate the problem to 'its between this box and this box' but was a pretty quick find after that :)

Something similar actually happened somewhere I worked.

There was an outlet in the hallway, right outside the glass window looking into the server room... you guessed it -- plug in a cleaning machine, blow the fuse, take down servers...

Our garage door opener has this problem. At very specific times of the year the sun confuses the infrared blockage sensor. The cause occurred to me when I lined up my eye to see what the sensor was seeing when it was failing, and I noticed the morning sun right next to the other end of the sensor stream. I moved a trash can to shade the sensor and it worked fine.

Just switch the sensors. Move the light source to the other side of the door, and the receiver to the shade. :tapstemplegif:

(That's how I fixed mine.)

Then the evening sun could potentially cause the same thing. There's a gap in houses across the street that may allow sun in at the right time of year.

I had the same problem with my garage door opener. Tried shading the sensor but it still didn't work reliably. Finally replaced both the sensor and transmitter and it has been good ever since. My theory is that the lens got dirty or scratched over the years and was picking up stray light from odd angles.

+1, I had this issue as well and went through a bunch of futile efforts to block the sun and the angle and wish I had replaced the transmitter a lot sooner instead of trying to move trash cans because that worked 50% of the time. I think this is the kit I bought: https://www.chamberlain.com/safety-sensor-kit/p/041A5034

On sunny days our garage door won’t open until the remote is very close. Either the solar panels or inverter generate enough noise to interfere with the signal.

A few months ago our garage door openers started working like normal again. That was great until we realized it was because the inverter had failed. When the inverter was replaced, our door problems started again.

Damn thats one electrically noisy inverter. I wonder how it passed EMC testing in the first place.

If I remember rightly, this incident was the motivating factor for adding "save failing images" capability to our software.

So much time and money wasted by not recording the camera. Could've simply reviewed the footage from the right timestamp and immediately discovered what was wrong. All you had to do was take a still picture every time the system makes a rejection.

Looks like a cool solution, when issue is known )

In hindsight everything is easier, sure.

But if you have a problem only happening sometimes, then you surely want all the data you can get from all sensors recorded, so looking at the saved video ofthe error time seems a nobrainer, but maybe it was not so easy to make them recording something. We do not know the setup.

> The technician rushed over to the equipment and discovered that the sun was shining at exactly the right angle to cause a lens flare in one of our cameras.

Somewhat infamously, "a rare alignment of sunlight on high-altitude clouds above North Dakota and the Molniya orbits of the satellites" the Soviets used for their nuclear attack early warning system triggered a false alarm, which, had it been treated as a real situation, could have lead to nuclear war in the early 80s.

https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...

This is something that I've also heared from hot axle box detectors for trains. Their solution: plant a tree.

Smort, trees are great for managing heat. I do hope they documented somewhere that the detector (or the thing it observes?) needs to be in shadow though.

Bughenge.

My exact thought. Please tell me this was filed as the Stonehenge Bug.

Being inside a building you can use Abu Simbel bug too.

preemptivly storing images from camera of rejected samples would have saved everyone more time, as it would be enough to review images of failed samples and notice a flare. of course if that was an option.

If I remember rightly, this incident was the motivating factor for adding "save failing images" capability to our software.

like stonehenge, but with industrial sensors

Industryhenge?

Amazing story!

Indiana Jones discovering the location of the Ark.

Reply on news.ycombinator.com