The Neural Net Tank Urban Legend | Hacker News Reader

The Neural Net Tank Urban Legend

gwern.net

105 pointsbelter48 comments

My friend once asked me to help him with an autonomous RC car race, where we’d go to a warehouse and train the car around the track.

After a few weekends of iterating and improving the model/training set, we were convinced we’d win the next race… only to lose almost immediately by crossing the lines on the track.

We did some inverse model explanation work which quickly showed that our car was paying attention to the overhead skylights more than the actual tracks. Unlike the other weekends, it was a foggy day!

A quick hack to cut out the top 50% of each training image brought our car back to its prior reliability.

In college, I had to make a line follower car that could drive over a line drawn with segments of black tape. While most folks just used a micro-controller to write their control loop, I made a set of op-amp circuits to compute the power to each wheel in analog.

It took a while to get good resistor values, but the night before the project was due I managed to get it working really well. So that was cool, and I got to catch up on sleep. Well the funny thing was that the next day we tested our line followers in this big lecture hall and the courses were not directly under the lights. So my beautiful analog line follower started following its own shadow!

Of course, the more diligent students had already tested their cars under these conditions and found the need to put a little hood over the photoresistors so that they only saw the light reflected from the car's own LED.

I’m going to bet that well-experienced lesson has positively affected future endeavours. Did it ever come to mind when you were working on something and the memory guided you towards searching for additional edge cases or taking “what ifs” more seriously?

Another good trick here is to do two measurements. Once with a light on your bot aimed at the line on, and once with it off. Taking the difference between these measurements will fairly reliably reject ambient lighting.

Of course getting that to work with all-analog hardware is left as an exercise to the reader.

> We did some inverse model explanation work which quickly showed that our car was paying attention to the overhead skylights more than the actual tracks.

It must've been posted here before: this is exactly the strategy Andy Sloane used to localize his car.

https://www.a1k0n.net/2021/01/22/indoor-localization.html

Ah very cool! Thanks for sharing.

We did this back in 2017(?) using an Nvidia Jetson TX2, allowing us to train the model on the car itself, avoiding a round-trip to our laptop!

A fictional variant of this story also shows up in Peter Watts' novel Starfish.

In that version, the neural net is trained to manage the ventilation system of an underground train station, based on data about passenger movements. Unfortunately, it ends up simply paying attention to the display of a clock that happens to be visible through one of its cameras. And when the clock breaks, a bunch of people asphyxiate. A bit implausible, but memorable.

Some time in the mid 90s I spoke with someone who had worked on this kind of problem (and others) on the London Underground. There had been a desire to know platform occupancy, and that this could be achieved by using a NN. The story as relayed to me was that the NN was never much more reliable that simply measuring the brightness of the platform — or rather the reflection of lights from the platform surface — which could be fairly easily calibrated

I took it less as a cautionary tale about NN and more about looking for simple solutions

My AI lecturer at University told both the tank story and a similar London Underground story. In his version of the latter, the NN was measuring platform occupancy but sometimes produced anomalous results. After investigation it was determined that the NN was actually counting the number of bins that were visible on the wall (pre-IRA, I guess). On average, the more people the fewer bins are visible.

Just going throught the trilogy - it's good! It's also free to download - go read it!

Content warning: the second book is 90s edgy to a degree uncommon today. Of the two major themes of the book, one is sexual assault.

Which, to be clear, is unambiguously bad both in the book and reality.

Sure, but the average person, on seeing just a "it's good!" review, might be surprised by the hundreds of pages describing the viewpoint character being raped.

Being OC I agree - I was personally shocked and had not read it before writing this "review"

It feels gratuitous and shocking, readers beware

By the same metric is The Handmaid's Tale "90s edgy" then?

I’m shocked that nobody has mentioned this yet: AI intended to recognize skin cancer instead learned to identify doctors’ markings and rulers.

https://jamanetwork.com/journals/jamadermatology/fullarticle...

https://venturebeat.com/business/when-ai-flags-the-ruler-not...

Yea this is what I was thinking of. Another work in that direction: https://www.nature.com/articles/s42256-021-00338-7

I never heard of the tanks story, but I vaguely remember stuff in radiology which was a similar story basically, it learned some spurious correlation to the label.

"This study suggests that the use of surgical skin markers should be avoided in dermoscopic images intended for analysis by a convolutional neural network."

Or you train with images where those markers are already removed by another NN? Or use some photos without cancer but with markers in the mix.

I heard a similar story about AI being used on MRI scans.

What the researchers hadn't realised was that the samples were coming from two different models of MRI machine and that patients with more serious symptoms / worse expected outcomes were more often being sent to be scanned on the more advanced of the two.

The AI had just detected some hidden difference between the outputs and was infering that the patients scanned on the expensive machine were more likely to have a serious condition.

Heh. I heard this in a cybernetics lecture in late ‘92 or early ‘93. I have never been able to track down the original either.

It's not the funniest tech urban legend story I've heard in a lecture -- that award would go to the kangaroos-with-grenade-launchers story -- but it is good.

Could you elaborate on that story or give a link?

Not a neural net story.

I only remember it hazily but it was just very funny in the moment; the lecturer was one of the most naturally funny people. It concerned as I recall it a combat flight (helicopter I think) simulator that had been unsuccessful and was being hastily tweaked as a demo for a civilian contract.

The story went that elements of the simulator were closely tied to ground attack scenarios, and that there was not time to remove all the code concerning enemy soldiers, so the models of the soldiers were replaced with models of kangaroos (same sort of height, walk on two legs).

On the day of the demo, there was some impromptu change to the script — they flew lower and too close to the ground or something — and the kangaroos fired at them with rocket propelled grenades.

I don’t believe the story is really true, but as a cautionary tale about reskinning demos it has always stuck with me.

https://www.snopes.com/fact-check/shoot-me-kangaroo-down-spo...

The debunking, for those not immediately inclined to follow the link:

> “...we had not set any weapon or projectile types, so what the kangaroos fired at us was in fact the default object for the simulation, which happened to be large multicoloured beachballs.”

Yeah, this would be the one I guess.

The lecture in question took place in late '94 or early '95, in the UK -- so four or five years before that telling.

I also came across this one when I worked in the sim industry (supporting army training). In my own experience, lots of sim errors could manifest in vaguely humorous ways, such as landing craft sticking vertically out of the beach when they landed, tanks whose turrets span rapidly, etc. These were usually obvious in testing. The more challenging errors occurred when multiple sims were federated together, since specific errors were often intermittent and might only be noticed when you could see the different screens together. Examples typically related to weapons effects and visual representations, e.g. platforms being invulnerable to missiles fired from a different sim, or a platform or person that looked different depending on which sim was displaying it. These issues were collectively referred to as 'fair fight' problems because it meant that trainees in one sim might have an advantage other others.

Disagree wholeheartedly with the conclusion here. Even though the story is untrue, sharing it is a fantastic way for students to grapple with a new concept and come to grips with the real problem of biased datasets. They'll remember it for their whole careers and share the wisdom with others too.

The death of mythology in the age of science has really done a disservice to mankind.

Interestingly, this story may predate the computer revolution - I am thinking about the anti-tank dogs: https://en.wikipedia.org/wiki/Anti-tank_dog

These were basically dogs strapped with bombs which were trained to run towards enemy tanks, where a contact detonator would cause the bomb to explode, killing the dog and hopefully causing damage to the enemy tank.

The analogy comes from stories about why the dogs failed in their main purpose - one story says that the dogs were trained on friendly tanks, not enemy tanks, so on the battlefield turned around and ran towards friendly tanks instead of the intended targets.

One could imagine this story was retold, and eventually re-interpreted as being as a result of an artificial neural network, rather than a biological one :-)

I wouldn't be surprised if the urban legend had been inspired by Feynman's famous anecdote of the mouse maze-running experiment [0] (recently discussed here, the link is also by the author of this post)

[0] https://gwern.net/maze

(edit: the earliest refs for the tank story predate Feynman's, so this would be wrong)

The Author and that site are ironically the polar opposite of any ai generated text output of NNs. These stories are the art of long form essays thoughtfully ideated and crafted over years. Amazing guy and finding something like this is the reason HN is awesome.

While this may be an urban legend, I have definitely played with ML and seen things overfit in comical ways.

It reminds me of a similar story about using deep learning on medical imagery for diagnosis. In one study, it supposedly learned the to tell which device was used to take the images. Apparently that was somewhat correlated with the results as more serious / heavily suspected cases were sent to a specific hospital in the region.

Similarly, I recall a talk by Daphne Koller where she talks about a heart attack predictor using X-rays as input. They found that stents were the biggest factor for positive prediction. Which is useless because the doctor would know if the patient had a stent.

A similar story I heard from colleagues in the field was about a CNN trained on images of tumors that ended up learning to identify images containing rulers that were placed there by a human for reference.

That's a big reason why the urban legend is so popular.

Clever Hans was a real comically smart horse!

https://en.wikipedia.org/wiki/Clever_Hans

>Clever Hans (German: der Kluge Hans; c. 1895 – c. 1916) was a horse that was claimed to have performed arithmetic and other intellectual tasks. After a formal investigation in 1907, psychologist Oskar Pfungst demonstrated that the horse was not actually performing these mental tasks, but was watching the reactions of his trainer. He discovered this artifact in the research methodology, wherein the horse was responding directly to involuntary cues in the body language of the human trainer, who was entirely unaware that he was providing such cues. In honour of Pfungst's study, the anomalous artifact has since been referred to as the Clever Hans effect and has continued to be important knowledge in the observer-expectancy effect and later studies in animal cognition. Pfungst was an assistant to German philosopher and psychologist Carl Stumpf, who incorporated the experience with Hans into his further work on animal psychology and his ideas on phenomenology.

NLP's Clever Hans Moment Has Arrived (thegradient.pub)

https://thegradient.pub/nlps-clever-hans-moment-has-arrived/

https://news.ycombinator.com/item?id=20861586

Finding and removing Clever Hans: Using explanation methods to debug and improve deep models:

https://www.sciencedirect.com/science/article/pii/S156625352...

Is your AI a “Clever Hans”?

https://medium.com/high-stakes-design/is-your-ai-a-clever-ha...

The Clever Hans Effect in Machine Learning: an overview by Bhusan Chettri

https://www.issuewire.com/the-clever-hans-effect-in-machine-...

Deep Learning, Meet Clever Hans

https://towardsdatascience.com/deep-learning-meet-clever-han...

Welcome to the cleverhans blog: This is a blog by Ian Goodfellow and Nicolas Papernot about security and privacy in machine learning.

http://www.cleverhans.io/

>This is a blog by Ian Goodfellow and Nicolas Papernot about security and privacy in machine learning. We jointly created cleverhans, an open-source library for benchmarking the vulnerability of machine learning models to adversarial examples. The blog gives us a way to informally share ideas about machine learning security and privacy that are not yet concrete enough for traditional academic publishing, and to share news and updates relevant to the cleverhans library.

While writing OP, I was curious enough about how they could solve Clever Hans that I went and read the original Pfungst book. I recommend it: his thoroughness & cleverness in studying Clever Hans (as well as the history of previous 'talking animals') ought to be better known than Clever Hans himself, IMO. (Brief review: https://gwern.net/review/book#clever-hans-pfungst-2011 )

Note that the author is saying that he couldn't find any proof that the story is true (e.g. a primary source or corroborating evidence) from his internet searches (Google, Hacker News etc.) But just because something is not on the internet doesn't mean it didn't happen. In this case, the story refers to a military (presumably classified) project from the 1960s (i.e. before the internet), so I wouldn't be too surprised if there was no proof on the internet.

Gwern,

If we're still around in 2030, you're gonna have to rewrite this same post about the AI drone that blew up its controller.

Here's hoping!

Oh wow, I read this story as evidence that neural nets are not the future back when I was an undergrad studying machine learning. Maybe I should have followed my instincts back then because even then neural nets seemed intuitively very interesting even if the statistics and math professors hated them because they weren't derived from any first principals.

Doesn't the derivative chain rule count?

Common element to these stories seems to be that the error is only found too late, and then apparently the whole project is scrapped. Most of the real projects I'm familiar with discovered some flaws in data or training, but continued working afterwards. Like nobody who heard this story ever thought to ask, so what happened after you trained it on some more photos?

I’m pretty sure I first heard this anecdote in Andrew Ng’s original machine learning course on Coursera.

In an AI class I took circa 1999 a professor recounted this story to us but it was “shadows” that the neural net detected.

The funny thing is he told us it was an urban legend but one that had a valid point.

I vividly remember hearing that story from a friend when I was in highschool around the mid 2000s.

Even if it's a myth, the moral is right.

>[...] with a probable origin in a speculative question in the 1960s by Edward Fredkin at an AI conference about some early NN research [...]

RIP Edward Fredkin, who recently passed away on June 13, 9 days ago.

https://en.wikipedia.org/wiki/Edward_Fredkin

>Edward Fredkin (born October 2, 1934, died June 13, 2023) was a distinguished career professor at Carnegie Mellon University (CMU), and an early pioneer of digital physics.

>Fredkin's primary contributions include work on reversible computing and cellular automata. While Konrad Zuse's book, Calculating Space (1969), mentioned the importance of reversible computation, the Fredkin gate represented the essential breakthrough. In recent work, he uses the term digital philosophy (DP).

>During his career, Fredkin has been a professor of computer science at the Massachusetts Institute of Technology, a Fairchild Distinguished Scholar at Caltech, and Research Professor of Physics at Boston University.

Ed Fredkin - Reversible Computing (Keynote from the CCC's Workshop on Reversible Computing)

https://www.youtube.com/watch?v=ROv1HX-gdas

Fredkin tells a great story about making Stephen Wolfram's eyes pop out of his head by showing him how to transform his own rule into a reversible rule.

Reversible Computing

https://en.wikipedia.org/wiki/Reversible_computing

Fredkin Gate

https://en.wikipedia.org/wiki/Fredkin_gate

Digital Physics

https://en.wikipedia.org/wiki/Digital_physics

Conservative Logic, by Edward Fredkin and Tommaso Toffoli

https://web.archive.org/web/20061017232512/http://www.digita...

Digital Philosophy

https://web.archive.org/web/20170729191558/http://www.digita...

Google Scholar: Edward Fredkin

https://scholar.google.com/citations?user=5QMmygwAAAAJ&hl=en

Rudy Rucker writes about his CAM-6 in the CelLab manual:

Cellular Automata Laboratory: Fourmilab home: Origins of CelLab: Classical Era: Von Neumann to Gosper

http://www.fourmilab.ch/cellab/manual/chap5.html

>Computer science is still so new that many of the people at the cutting edge have come from other fields. Though Toffoli holds degrees in physics and computer science, Bennett's Ph.D. is in physical chemistry. And twenty-nine year old Margolus is still a graduate student in physics, his dissertation delayed by the work of inventing, with Toffoli, the CAM-6 Cellular Automaton Machine.

>After watching the CAM in operation at Margolus's office, I am sure the thing will be a hit. Just as the Moog synthesizer changed the sound of music, cellular automata will change the look of video.

>I tell this to Toffoli and Margolus, and they look unconcerned. What they care most deeply about is science, about Edward Fredkin's vision of explaining the world in terms of cellular automata and information mechanics. Margolus talks about computer hackers, and how a successful program is called “a good hack.” As the unbelievably bizarre cellular automata images flash by on his screen, Margolus leans back in his chair and smiles slyly. And then he tells me his conception of the world we live in.

>“The universe is a good hack.”

[...]

ON THE SOUL: Ed Fredkin, Unpublished Manuscript

http://www.digitalphilosophy.org/wp-content/uploads/2015/07/...

>The John Cocke Theory of Dreams was told to me, on the phone, late one night back in the early 1960’s. John’s complete description was contained in a very short conversation approximately as follows:

>“Hey Ed. You know about optimal encoding, right?”

>“Yup.”

>“Say the way we remember things is using a lossy optimal encoding scheme; you’d get efficient use of memory, huh?”

>“Uh huh.” “Well the decoding could take into account recent memories and sensory inputs, like sounds being heard, right?”

>“Sure!”

>“Well, if when you’re asleep, the decoder is decoding random bits (digital noise) mixed in with a few sensory inputs and taking into account recent memories and stuff like that, the output of the decoder would be a dream; huh?”

>I was stunned.

tldr: "I think it's probably an urban legend, because whatever."

Reply on news.ycombinator.com