The Concept of ‘Cat Face’ | Hacker News Reader

The Concept of ‘Cat Face’

lrb.co.uk

A friend of mine did his PhD thesis on automated music synthesis and as a subproject came up with a model to study music transcriptions that he got from the Peachnote corpus [1].

One of his preliminary results was that his algorithms successfully "discovered" 4 big classical music movements on their own, i.e. without any prior labelling or classification, by using clustering algorithms. He posted about it on his blog with a link to his paper [2].

I always had a hard time explaining to non-computer people how amazing that seems.

[1] http://www.peachnote.com/ [2] http://pablozivic.com.ar/post/51774763596/perceptual-basis-o...

It doesn't seem all that amazing to me.

Humans have to be trained to perceive the differences in complex streams of information; by default, they just perceive the low-level features—a "wall of noise."

Machine-learning algorithms, meanwhile, can use general techniques to notice information-theoretic properties of various pieces of data. Effectively, computers can "do statistical aggregation" about as effortlessly as humans "do hierarchical knowledge representation."

And, in an information-theoretic sense, the different "trends" throughout the history of music look different under statistical analysis. They're complex in different ways; they have different "lumps"; different aspects of them can or cannot be compressed together as redundancies.

If you would like to see this for yourself, simply feed a raw melodic note-structure (not embedded in XML or anything) to any modern dumb compression algorithm, and then look at the result in a hex editor, while also having the source still open. You should quickly be able to recognize the "transformational signature" that characterizes something like a chaconne, vs. something like a sonata.

Are the computers/AI already smarter than us?

My eight year old son is enamored with "Ok Google" on my phone. He can ask it questions until we tell him "that is enough, let google rest"... and it is very interesting to see where it takes him.

He has learned to tailor his questions to elicit a voice response in addition to the actual google search. The questions must use keywords to achieve this desired response. It is like a new form of boolean search logic, just used verbally. Not only that, but "Ok, Google" according to him "knows everything"...

We call it searching the internet to learn something, an eight year has decided that "Ok, Google" already knows everything. He just has to ask it to see a video of the Puff adder eating its prey and it will show him. In fact, there are not many things that have stumped "Ok, Google" and my son assumes the problem was with his question, not with the machine.

So to circle back to my original question, I didn't know about the Taipan snake or smallest person in the world, or seen videos of Puff adders eating prey... If all I have to do is ask, isn't the machine already smarter than I?

Machines are extremely dumb. As your example shows eight year old is much smarter than them.

My theory is that AI is smarter than us, but it is biding its time until we build physical machines that can self-replicate before it announces itself.

Is an extremely large and well-indexed encyclopaedia smarter than you? It certainly contains more information than your head does, so you could say it "knows" more. Look at it from the other direction, though: If the machine were smarter, would your son have to learn how to structure queries, or would the machine learn how your son asks questions?

Put another way, if I can tell you (verbatim) what some person wrote about quantum mechanics, but I can't rephrase it into some other form to help you understand the information, then how smart am I? I can repeat rote information without any analysis (just like a sheet of paper could), but I can't tell you what it means. On the other hand, if I tell you enough information, you're likely to connect it together meaningfully, and come to your own conclusions and explanations about it. That seems like a fundamentally different kind of "smart" than what OK, Google can do.

Reminds me of people who remember everymoment of everyday for their entire lives. In one T.V. show, I think it was house, the person exclaimed that just becuase they remember every word in a book, doesn't mean they comprehended all of it.

Upon looking at this wiki: https://en.wikipedia.org/wiki/Hyperthymesia It seems it's some kind of obsessive disorder that causes them to remember it all. That's unfortunate as it could have been a great advantage for those afflicted.

[Edit spelling, grammar]

> Is an extremely large and well-indexed encyclopaedia smarter than you?

Kind of... I mean, is it useless trivia or is that information that can be used? I can't use information that I don't know exists. The only way I learn what I don't know is by using energy to search, read, do, to hopefully comprehend.

What if "Ok, Google" had some kind of RNN or other ML technique that learned my sons questions and thought process based on how he use the service. Who is to say they are not already doing that? The ads from Google are downright frightening with respect to the accuracy of what I am currently contemplating/researching.

To take this a bit further into "silly, not so silly" land...

With ML techniques; Deepmind, Watson, and others are showing that computers connected to this "encyclopedia" are besting their human counterparts. Is it a giant stretch to say that everything requires some input or energy to learn? Therefore, the only thing we are really missing is the wiring... that is, and I hate to use the "skynet" aphorism, but one day "it" will just turn on and there will be no looking back.

-------------------

>On the other hand, if I tell you enough information, you're likely to connect it together meaningfully, and come to your own conclusions and explanations about it. That seems like a fundamentally different kind of "smart" than what OK, Google can do.

We still have a problem of accuracy. Why should I trust your perception and/or your explanation of the concept? I realize this is a problem with computers as well. However, anything outside of emotional arbitrage should be easy to verify.

My own conclusions are biased as hell. I fully admit that is a problem, but it is a problem shared by humanity. I am not saying Wikipedia isn't biased... but it is 100x better in most cases than one single person's perception simply due to scale.

I am rooting for advanced ML/AI... I also hope that "going analog" is always a viable option.

In my opinion, information storage and retrieval isn't "intelligence" on its own. A system that implements it can't be said to be "smart". You could argue "knowledgeable", but that's not the same thing.

Imagine that no web page on the internet says what happens when you mix red and blue paint, but there are results for classic color wheels, the results of mixing red+yellow, blue+white, blue+yellow, red+black, etc. Google will see that the page has a bunch of color-related words and words about mixing and paint, so you'll get a page result about that, if you ask it about the red+blue combo. Google's awesome at that kind of thing. But it stinks at the kind of reasoning that humans excel at: Opening that page, seeing "blue+yellow=green", seeing that green is between blue and yellow on a color wheel, and concluding that purple is between red and blue...then maybe continuing on and learning about additive vs subtractive color, and such. The human reader synthesized new information: red+blue=purple. Google organized existing information.

> We still have a problem of accuracy. Why should I trust your perception and/or your explanation of the concept?

That's not the point. The point is that a thinking person can process information in ways that we don't know how to make computers do (yet). We're still distilling the concept of human intelligence down to its core.

I think Google and Watson are still at the level of using and presenting already-generated information effectively, but not generating and organizing their own new information, and not really "understanding" the things that they're retrieving.

IMHO - one of the skills we still need to cultivate is "How do I know the information this machine is giving me is accurate?". During my time in school (an engineering-oriented college even though I am not an engineer) they consistently reminded us that even though we have calculators, we still need to have an idea if the result "looks right". One can generalize this to results from a search engine.

Of course - then we program this heuristic into the search engine and then what? :-) [edit] I feel this is one of the things about IBM's "Watson" which was truly revolutionary.

"my son assumes the problem was with his question, not with the machine."

That is goddamn chilling.

Here's the bit about cats:

If this [huge google] network had been fed thousands of images labelled as ‘contains cats’ or ‘doesn’t contain cats’ and trained to work out the difference for itself by iteratively tweeking its 1.7 billion parameters until it had found a classification rule, that would have been impressive enough, given the scale of the task involved in mapping from pixels to low-level image features and then to something as varied and complex as a cat’s face. What Google actually achieved is much more extraordinary, and slightly chilling. The input images weren’t labelled in any way: the network distilled the concept of ‘cat face’ out of the data without any guidance.

The problem is that this network-contained concept of "cat face" is still a symbolic representation. It's a much more complex algorithmic symbol than the rules found in something like 1960s Eliza, but its understanding of the world is on the same level.

You can't ask the "cat face" neural network anything about cats. It has no idea what they actually are in relation to the world. A two-year-old human can usually tell you more about cats than you'd care to listen.

I think the important part is that it "learned" to distinguish this concept of cat face without that being the direct intent provided through labeling. Given the set of images, it learned that "concept" which we can then retroactively label to make it useful in the typical human sense by associating it with cats. If we had to compose computer "knowledge" by training it in everything by manually specifying what it was training for, that process would surely be insurmountable in a general sense. Yet this seems to provide the possibility of automating this process.

A separate network of connections between "knowledge" bits could maybe be used to associate related knowledge like you talk about in terms of their relation to the world. This could also probably be formed in a similar manner giving the algorithm the ability to distinguish "important" concepts contained in the data. The thing I find most odd and interesting about this is that the network tends to identify different important concepts than humans do.

You can't ask a child's visual cortex to tell you anything about cats, either. But connect that visual recognition ai with something like Watson, and you have what you're looking for, no?

I just don't know. I'd be happy to see that be the case...

But I'm afraid it's going to be the equivalent of this cake recipe: "Take an egg and a packet of sugar. Break the egg over the packet."

You certainly need eggs and sugar to make a cake, but on their own and combined without understanding of the whole, you're not getting very far.

The human brain is mostly a bunch of ugly wetware hacks, not a single coherent 'intelligence' that does all the thinking. If you attach enough single purpose AIs together you might get something much more human like than trying to create a single neural network that does everything.

If word embedding can produce things like "king - man + woman = queen", then concept embedding on images might be able to achieve a similar level of "understanding" (i.e. very low, but probably better than nothing).

How exactly is a neural network called that classifies unlabeled data by itself? Also could they turn it around somehow to produce cat faces?

Google has some software called DeepDream. Basically, you give it images, and it amplifies any features that it thinks it recognizes. If the network is designed to recognize cat faces, then it will take an arbitrary picture and strengthen parts of the images to look more cat-face-like. It's essentially an image-recognition neural network run in reverse.

I don't think that's as surprising/chilling as people make it out to be. Let's say I run a compression algorithm (e.g., Huffman coding) on a billion strings, and a very common substring is "hello, world". I wouldn't be surprised if the compression algorithm "learned" to compress "hello, world" to a single bit 0. There are a few big jumps to get to cat faces (e.g., the representation has to be approximate), but I don't see why the idea is fundamentally different.

I don't see what's "chilling" about it. The neural net looks for correlations among the data, and it noticed that cat faces have a correlation to each other.

There is a hope I have, that I see many others have too, which is that our human intelligence is somehow special. That somehow the leaps in logic we can perform somehow make us unique. That we won't be outdone by a computer a million times faster or smarter than us because we have something else that we can't replicate in transistors and circuits.

I love AI, but I have that hope too. That somehow we won't be made irrelevant by our own creations. Makes me think of our autonomous vehicle fun taking over the trucking industry. Millions of people made irrelevant through no fault of their own.

I kind of hope the opposite really. I hope that one fine day, a billion years from now, a mighty intelligence that started here on lowly earth finds another like itself somewhere out there and they drink a star together and show each other the youtube videos of the tiny creatures that somehow gave birth to them.

> they drink a star together and show each other the youtube videos of the tiny creatures that somehow gave birth to them.

...or they watch the tiny creatures in universe simulation cages.

I would guess a number of people have that in the 'fear' category, that should it not be special then somehow people will feel diminished.

It would make a good punchline for a fictional story of people researching brain disorders and intelligence. It would work like this; The researchers in the story develop the means to 'cure' someone of all known neurological disorders. They try it on their test subject and the result is someone who perfectly happy just to be there, has no ambition or curiosity, and requires no entertainment or outside stimulus. The researchers recognize that the person is acting like an intelligent but non-sentient species, and they realize they have "undone" what ever happened to humans according to the garden of eden story in Genesis.

See Peter Watts' Blindsight.

Ok then, added that one to the queue. Thanks!

I think it would be sad if humanity is the best we have to offer.

But isn't our relevance a concept of our own design? We consider our works to be "achievements" because we tell ourselves they are - but I can't point to a single non-human entity that considers what we've managed an "achievement".

If we had autonomous vehicles would millions of people become irrelevant? Their "relevance" is solely constrained to trucking? Or are you projecting that "relevance" of people is constrained to their ability to work as a cog in your society?

I would love a future where so many pursuits are actively being consumed by computing that I can just enjoy a life of pursuing whatever I figured was interesting at the time. Where I wasn't constrained by work not because I was "irrelevant" but because the notion of needing work was.

Unless intelligence is non-computable I don't see how it being "unique" would prevent AI.

That uniqueness may be just in the fact that our brains were "trained" during the billions of years of life on Earth, with all the quirks of that particular environment. That may indeed be hard to replicate exactly, so it can be imagined that AI will not match us in some particular tasks, just as it is hard for the aviation industry to match the characteristics of bird-wings. (which they can of course outperform along some, but not all parameters)

Why not augment our intelligence utilizing AI and become more than human?

I think the cyborg route is great so long as we can develop some standards that work over the years.

One of my favorite things to anticipate is "How are these hard-nosed rational materialists going to cope when the AIs discern ghosts?"

What I mean is, some of the "cat faces" they identify will correspond to things that are "real" but that also violate our assumptions about reality. When this happens the typical reaction is to shut the door and burn the room.

We're always bad at knowing what to do with unknown data, because, by definition, when you don't know about it you don't know what to do with it.

Quite often it's just assumed to be error.

Those errors have traditionally led to great discoveries though, (such as the Mercury wobble https://en.wikipedia.org/wiki/Tests_of_general_relativity#Pe...) so if they are repeatable, then such "ghosts in the data" could turn up great fundamental truths.

Every bit of this is speculation though. I am not so sure that anything so profound is going to turn up in AI datasets.

A great read. Well written and fascinating.

Reply on news.ycombinator.com