Hacker News Reader

The funny thing is that even the easiest games are surprisingly difficult, even insurmountable, for otherwise capable humans who have not been "trained" on what to expect and learned domain knowledge through trial and error. See [0].

[0] https://www.youtube.com/watch?v=ax7f3JZJHSw

You're arguing against a position that nobody sensible holds. It's like you're yelling at the invention of the train because it can't fly to the moon. Well, sure, but it's a necessary step on the way. Researchers aren't not solving your problem because they've not realized it would be an impressive thing to do, they're not solving your problem because the science isn't ready to tackle it yet (and also because it's an incredibly biased metric).

This[1] paper does an excellent job gradually removing cultural priors from a simple videogame, and measuring the extent to which human and computer performance degrade. Their conclusion is that human ability to learn games fast depends on our ability to map them to real world concepts. Would it be permitted for entrants in your contest to pretrain on some other non-videogame task?

[1] https://arxiv.org/pdf/1802.10217.pdf

I didn’t state it very clearly but the whole point of my test is the ability to translate cultural priors into a new medium.

Here’s another task I’d like to see: have a kid who has never seen a film before read a children’s book and then watch the film adaptation. Test the child on their ability to identify all of the characters in the film. This is the sort of task any kid could do with high accuracy but an AI would struggle with unless it were specifically trained on a corpus of books and their film adaptations.

But like the parent said, you still have to make it fair. The child may have never seen a film before, but they've probably read other books, or have had books read to them, or have had stories told to them (real or fictional). And they've probably seen TV shows or videos of some sort. And they've probably had an upbringing with lots of adults and kids speaking to and around them about topics, and have had (rudimentary) experience telling stories themselves, like what they did today. All of this helps them form a model of the world that makes understanding a book or a film, and the relation between them, easier.

So maybe training on a corpus of books and film adaptations is cheating in this scenario, but the AI should still be allowed to train on a massive corpus of all of the things a child may be exposed to. Maybe instead of showing it films, show it thousands of hours of footage from head-mounted cameras that people have worn while going through daily life, and children's TV shows, books, and games, and have it talk to a bunch of humans.

Here’s my take on why it’s not intelligence: The hardware has no capacity to independently observe itself or the conditions that give rise to its existence, and given its observations, improvise as needed.

Everything we would label without question as possessing intelligence has an instinctual drive and mechanical ability to improve itself, to beat itself at its own games. To move out of the way of danger if it has the opportunity to see it coming. To improve the probability of its survival.

There’s more to intelligence than building a machine of a predetermined spec to mechanically sort a predetermined set of data. These notions in comp sci seem to completely ignore physical reality. I can conceive of a planet called Krypton, can it exist? I can conceive things like tautology but can physical reality provide an example? I can conceive of a phone app giving me a sense it has intelligence but it doesn’t. It’s not self aware and cannot self replicate. It can’t save itself or beat the odds on its own.

This physical reality we occupy invented what we’d define as intelligence. Our definition ought to very specifically include physical capabilities of independent not programmed self awareness & capacity for self care. AFAIK, that’s what our basis for the concept of intelligence is based on. Why should we accept an artificial intelligence that’s less fully capable of that as somehow on par or better than is given its obvious limitations?

I agree. But I think it's a long and gradual path to eventually reach that stage. This would just be one of the many stepping stones before true intelligence may be within reach.

Humans do have lots and lots of prior training albeit not specific to that exact task but I still understand your overall point. The more generalized the AI the stronger the claim of being artificial intelligence.

I'd be interested in a system that can beat arbitrary games that are considered relatively simple, or complex across a genre. I'd like an AI that can play bejeweled and then easily transfer that information to an arbitrary merge game, or an AI that can play CoD and transfer that to Overwatch, or an AI that can play MapleStory and then transfer that information to Mario brothers. Or an AI that plays League of Legends and transfers that knowledge to DotA.

Does such a thing exist?

DeepMind's AlphaZero?

I think it's an incoherent premise to suggest that you could have a human with no contextual knowledge or experience that pertains to playing an arbitrary game. There's no blank-slate human that you could test like that. Even the act of finding food and eating, walking around and avoiding hazards, observing the movement and behavior of other people and animals..those experiences shape a person's ability to interpret an environment and model the agents within it. Those priors would be useful for any first-attempt at a game.

AlphaGo needs to be retrained for new games. The breakthrough was in having a generic training algorithm that could be set loose on many games and learn them.

Oh, feel free to have your AI read all the books and watch all the films you want. Heck, you can even let your system read the game’s manual, just as a kid would do if they got stuck in the game. The only restriction is that it can’t play any video games as training.

The point of my challenge is to demonstrate understanding by being able to translate cultural and contextual information into a new medium. So rather than just making connections between different texts, you need to be able to take a text and leverage it to understand what’s happening on the screen in a video game.

Maybe you've already learned this from the replies, but it's not really possible to have an intellectual dialog about the meaning or limitations of this research.

So while I get what you're saying and I'm sympathetic to your point of view, it's too late. The cloud bucks and 5x VC multiples that can be made on this stuff is too good. Even academics, whose whole advantage is that they can resist market forces, have very lucrative careers awaiting them if they publish the right stuff.

Then there's all the adjacent benefits. Like what good does it to be publicly skeptical of DeepMind? Google is everywhere in tech and science. Or even some random guy's paper, like where is he going to work? I can count the number of opportunities on my hand, and they all have a lot of influence in technology broadly. You don't even have to be doing AI for it to be rational to simply say, "Wow what an Amazing Discovery" and move along.

There's literally no forum that profits from skepticism. Nobody wins when "AI," whatever it is attached to, loses.

Hm I don't know - I'm pretty bad at games and will try hard at Pong at still lose, my coordination is poor. Maybe you need to be more specific, that the AI can learn the game mechanics without training?

I do not believe that game exists. There’s too much embedded knowledge w/r/t playing video games. I am happy to be proven wrong; did you have an example in mind?

For a concrete example, I'll nominate the very first video game I ever played: The Legend of Zelda for the NES. This is by no means an easy game, however, so I'd alter the win condition: you don't need to beat the whole game, just collect a single piece of the Triforce.

As a kid I was able to gather the first 6 pieces of the Triforce without any outside help. I think an AI should be able to get at least one, given that it's playing a video game for the first time.

As for the "embedded knowledge", I'm happy to talk about context in terms of cultural references, heroes, the monomyth, etc. The Zelda series draws heavily on these concepts, though I am not sure how much I knew of them as a 6-year-old.

I'd say a really simple game like Pong or Snake would already be enough. Only two(or four) possible inputs, simple graphics.

The difficulty could also be decreased, but in the end the challenge would be to grasp how the game works, not to beat highscores.

With a general enough measure you’ll have created general intelligence. e.g. for humans, quantity of reproduction

Sure but humans have been exposed to the controllers, sounds and all these abstractions from previous games. Your AI should have the same and fair exposure. You are looking for a general game-player AI.

tl;dr: Your brain is training 24/7 non-stop.

Please reread my comment. I specifically state that this test is only to consider a game easy when it is the first game a human has ever played. So the problem of humans having experience with other video games is not present.

So where do you get the data on how well humans did on this hypothetical video game, when that game is the first video game they ever played in their life?

It's doubtful whether such data exists. Even if it did it's questionable whether such results have any bearing on human understanding or intelligence.

> first game a human has ever played

Like a human that never played a game, used a computer, used any physical buttons, learned a language, understands what a game is, knows what a sport like football is, etc...

There is lots of stuff that you'll need to feed your AI to be equivalent to a 6 year-old brain. That and a 6 year-old interaction with a game will be more random than logically driven.

If you are doing that you are already at the stage of AGI. It is not theoretically impossible but the impracticality of it right now is what makes scientist trying domain-specific AI.

There is lots of stuff that you'll need to feed your AI to be equivalent to a 6 year-old brain... If you are doing that you are already at the stage of AGI.

That’s kind of my point. The article is asking the question “do they understand?” and my answer is yes, if you can do this. Read, to a 6-year-old, a kid’s book about a knight with sword and shield rescuing a princess and they will have no problem playing Zelda. Does the child understand the damsel in distress trope? I would argue yes.

This sort of test would be extremely difficult to get an AI to pass. Heck, it doesn’t even have to be video games. Just do any test that involves translating a concept from one medium to another (text and video). It’s extremely simple for humans to do and extremely hard for an AI.