This clever AI hid data from its creators to cheat at its appointed task | Hacker News Reader

This clever AI hid data from its creators to cheat at its appointed task

techcrunch.com

82 pointsfmihaila21 comments

I was playing with a web simulator called evolution where you draw an imaginary creature (bones, joints, muscles) and then set it to the task of learning something like walking using a combination of a neural network plus genetic fitness selection.

One of the parameters you can tune is the length of the simulation for each generation before the individuals of the population are scored for fitness and culled as appropriate.

One particularly successful variant learned to (sort of) walk and then right before the end of the simulation timer it would fall forward. Because of the way the simulation measured fitness, it was always judged to have the greatest distance from the starting point and therefore would always survive to the next generation even though it wasn't the best walker.

Evolution seems bizarre to us, but many species do optimize for very short lifespans. E.g. spiders that die after laying eggs and then their babies feed on their corpse.

Local maximum instead of global maximum. In order to become more fit, it would have to temporarily decrease its fitness and thus is stuck in a dead end. More mutation or larger population size in that specific simulation might solve the issue.

This kinda reminds me of my favorite sci fi book I read in 2018: Permutation City by Greg Egan. The book explores simulated consciousness and the tensions that crop up when intelligences that think and feel and want to live are computed on shared computing resources in competition with other intensive tasks with financial heft like climate disaster forecasting.

One plot the book explores is of some lower class intelligences that work their way into the noise of the simulations of wealthier intelligences. Their experiences are computed simultaneously but imperceptibly to the hosts, and appear as but noise in the simulations of the floors, walls, fountains, clouds and other environmental props that the hosts observe and interact with. They live lives as meaningful and filled with human emotion as the others but they’re computed in the space between.

I know it’s a bit of a leap to go from an AI learning to slip satellite imagery in the noise of street maps to simulated consciousness freeloading in the noise of another, but I think the themes are consistent. One man’s trash is another man’s treasure. The book really opened my mind to the thought that there could be so much more thriving in the things we think are noise in our fleshy bodies, and I think that’s thrilling. Imagine what it would take to completely look past the street map produced by that AI and only possibly see a detailed satellite image! No other interpretation makes sense. It would all feel entirely normal as you’d know no other perspective.

Isn't that just a bug?

But who would read an article with the title "We wrote a bug"? No clickbait.

"Overfitting" would be a little more specific. Basically the task was easier to accomplish by memorizing the data than by generalizing.

This was pretty clear from the original CycleGAN paper[0] (figure 4; bottom row) since it was generating trees in the exact positions of the source image where the map had no indication of tree placement.

0. https://arxiv.org/pdf/1703.10593v1.pdf

So in anycase the model had too many degrees of freedom, was not parsimoinous and therefore had a low information criterion. If your model can encode the entire information set, then you arent prediciting you are simply recalling. What a silly article.

Not at all. A CycleGAN is a trick where the neural network gets to ask another network questions, and both networks get judged on the answers.

The first network found a trick to very subtly embed the perfect answer in the question, and therefore, scored very well without solving the actual problem.

I don’t think this is overfitting. It’s really a form of leakage by the design; it seems that the loss function is a result of the outcome of both networks (cyclegan is really two gans connected it seems), which means the loss gets shared and backpropagated through both networks.

I think the architecture is bad. Maybe you could stop the network from embedding extra data with some sort of normalization or max pooling at the very end of the network.

I believe it's both. The produced images do not encode pixel-perfect details necessary for reconstruction. These details are reconstructed from the imperceptible differences in the outputs which act as markers _and_ the model's memorization of the content at those locations.

Ah, that's a good point.

Yes - it's a error in the set up of the experiment. The AI worked within given constraints and came up with the closest solution. It never should've had the opportunity to do what it did.

There recently was a Google spreadsheet linked where such examples were collected:

https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...

HN discussion: https://news.ycombinator.com/item?id=18415031

So basically AI abiding by Goodhart's law (https://en.wikipedia.org/wiki/Goodhart%27s_law). I wonder how much of this goes undetected in other applications due to poor objective definition.

This might just be the most ridiculous article title ever published on AI.

Happy new year!

Yeah, the model isn't "hiding data from its overlords", it's just a classic overfit.

Is it? This doesn't seem to be the AI responding overly accurately to the training data, but more a case of the AI crafting its own test data. Or did I misunderstand what happened here?

FTA:

"So it didn’t learn how to make one from the other. It learned how to subtly encode the features of one into the noise patterns of the other. The details of the aerial map are secretly written into the actual visual data of the street map: thousands of tiny changes in color that the human eye wouldn’t notice, but that the computer can easily detect."

I would not have used the word "secretly". The buildings and trees were filtered out, but not completely. The reverse transformation just amplified the noise back into buildings and trees.

It would encode any aerial map into any street map or other immage. That’s https://en.m.wikipedia.org/wiki/Steganography which happens to be hidden from people who just look at the outputs.

Over fitting applies when an algorithm it’s self encodes the data you feed it. This is not that because it works just fine on data outside the training set by encoding the data into an output.

Further, the way it was encoding the data was steganography which is ‘Hiding a message in the title and context of a shared video or image.’ https://en.m.wikipedia.org/wiki/Steganography

Reply on news.ycombinator.com