And yes you can fix the bug but the bike wheel guy shows you there will always be another bug. We need a paper/proof that invents a process that can put an AI-supported (non human intervention) finite cap or limiter or something on the possible bug surface
If anything, AI could help by "understanding" the real objective, so we don't have to code these simplified goals that ML models end up gaming no?
Yes, I have an intuition that this is NP hard though
I feel that a good first step would be to introduce some kind of random jitter into the simulation. Like, in case of the wheels, introduce road bumps, and perhaps start each run by simulating dropping the wheel from a short distance. This should quickly weed out "too clever" solutions - as long as the jitter is random enough, so RL won't pick up on it and start to exploit its non-randomness.
Speaking of road bumps: there is no such thing in reality as a perfectly flat road; if the wheel simulator is just rolling wheels on mathematically perfect roads, that's a big deviation from reality - precisely the kind that allows for "hacky" solutions that are not possible in the real world.
Feels halting problem-esque.
So to the OP's example "optimise a bike wheel", technically an AI should be able to understand whether a proposed wheel is good or not, in a similar way to a human.
We do understand the "real objectives", and our inability to communicate this understanding to hill-climbing algorithms is a sign of the depth of our understanding. There's no reason to believe that anything we yet call "AI" is capable of translating our understanding into a form that, magically, makes the hill-climbing algorithm output the correct answer.
Conglomerate developed an AI and vision system that you could hook up to your Anti-aircraft systems to eliminate any chance of friendly fire. DARPA and the Pentagon went wild, pushing the system through test so they could get to the live demonstration.
They hook up a live and load up dummy rounds system, fly a few friendly planes over and everything looks good however when they fly a captured Mig-21 over the system fails to respond. The Brass is upset and the engineers are all scratching their heads trying to figure out what is going on but as the sun sets the system lights up, trying to shoot down anything in the sky.
They quickly shut down the system and do a postmortem, in the review they find that all the training data for friendly planes are perfect weather, blue sky overflights and all the training data for the enemy are nighttime/ low light pictures. The AI determined that anything fling during the day is friendly and anything at night is terminate with extreme prejudiced.
like they're just not that many pictures of this stuff. we needed hundreds, ideally thousands, and had, maybe, a dozen or so.
okay, so we'll get a couple of talented picture / design guys from the UI teams to come out and do a little photoshop of the images. take some of the existing ones, play with photoshop, make a couple of similar-but-not-quite-the-same ones, and then hack those in a few ways. load those into the ML and tell em they're targets and to flag on those, etc. etc.
took a week or two, no dramas, early results were promising. then it just started failing.
turns out we ran into issues with two (2) pixels, black pixels against a background of darker black shades, that the human eye basically didn't see or notice; these were artifacts from photoshopping, and then re-using parts of a previous image multiple times. the ML started determining that 51% or more of the photos had those 2 pixels in there, and that photos lacking those -- even when painfully obvious to the naked eye -- were fails.
like, zooming in at it directly you're like yea, okay, those pixels might be different, but otherwise you'd never see it. thankfully output highlighting flagged it reasonably quickly but still took 2-3 weeks to nail down the issue.