Hacker News Reader

Tranfer learning is a form of this, where you train the first half of a neural net over a huge set of data, then take a much more specific task and train just a bit more by adding some layers, and you get great sometimes industry leading results. Look up Google's BERT project, trained generically on English texts and then you can reuse that part in many different tasks like sentiment analysis or entity recognition or translation. An amazing way to leverage Googles compute and data access and do cool things even as a single developer.

> as a single developer

I think you are out of luck already. Even with transfer learning, retraining BERT to fit a larger dataset takes ages even on an RTX 8000. A single state-of-art experiment/training run at Facebook now costs 7 figures. The age of individual SOTA ML researcher/developer is over.

This is so sad, actually, and It's not just ML, it is every kind of new tech and discovery.

To me it shows we have already picked the easy fruits and technology will not continue to evolve exponentially.

> In a strong sense, this implies ML is not "intelligent" in any conventional sense of the word.

I don’t know why people have so much faith in ML and AI. Considering what a CPU actually does, shouldn’t the default position be one of skepticism?

It’d be truly amazing if you can make a machine intelligent in the human sense by merely coming up with a clever algorithm.

Or we could flip your question around:

Considering what a neuron actually does, shouldn't the default position be that the brain is a computer?

How certain are we of what neurons and other building blocks of the brain actually do?

No, because you’re being reductive. Biological organisms are clearly not like silicon computers.

Maybe biologist come up with the next big break through. I mean, if you want to match an organically grown organism ...

Is it clear that they're not alike? In the end it's all information and data being processed, whether by silicon or neural cell. What difference does the medium make to the end result?

Then at the very least, you should be able to simulate a biological brain, neuron by neuron, and get an intelligent computer that way. If you agree that's possible, you also have to consider whether it's the most efficient approach -- plausibly special algorithms could be invented that get the same intelligence at a fraction of the computing power.

Every advanced tech has been used as a metaphor for how the mind works. People used to think the brain worked like a piston engine. Now people think the brain is like a computer. Some people also think we are a network ala the internet of biological persons. Lots of artists think in images. Some people see the brain as an evolving story. The people who have any idea how the brain works are those that have been studying it for decades, I would be interested in those metaphors.

The conversation around artificial intelligence is more of a call to action for human intelligence to figure out what we need to do and how wield ML than it is a description of the fundamentals of the computing. Even if the results are mediocre in the extreme, you can bunch together a huge amount of social and business change under that banner. I'd argue that's why people have faith in AI not because of what a computer can do, but because it is a call to faith in humans ourselves to integrate the technology and all implicated changes to it's maximum potential.

They do build on previous knowledge. That’s literally what the training is doing.

It’s just easier, engineering wise, to start over from scratch on each iteration. And it makes it easier to evaluate how each change improves the outcome or not.

But they do build on previous knowledge (that's the training part), it's just the such knowledge is built from scratch, rather than taught by others.

A human can learn something through a fairly small amount of specific knowledge via written, verbal, etc instruction.

All the data and the training process together are very clunky in comparison.

IE, an inability to be taught by others is a serious limitation to say the least.

Perhaps we can learn from small data, perhaps that’s an illusion. How much of our mind is hard-wired by our genes, our genes being a very slow-learning form of intelligence?

Or, when we see “one” example of a new animal (for example), do we see a single picture? Or do we see it moving for about a second? Effectively multiple images? And, having previously learned a general model for physics, stereoscopic vision, and lighting, does that one second of motion gives us the size, shape, and a guess at the musculature of the animal?

If you strip those advantages, how many training examples do we need? The examples of people blind from birth or from extreme youth who regain vision in adulthood[0] implies that we learn a lot in our youth about how to see, which we can share between recognition tasks we learn later. Keeping those basic skills — avoiding “catastrophic forgetting” — still seems to be a rare and advanced feature in AI, but it’s not impossible.

There is still a lot we don’t know about how our own minds work. We might be closer than we realise, or much further.

[0] https://en.m.wikipedia.org/wiki/Mike_May_(skier) amongst others. I never did find the name of the guy who learned to see from touching a monkey statue…

It's certainly worth keeping in mind the limitations that you describe and human's learning activity is certainly not as the rationalist intuition. But to take these limitation as entirely characterizing human activity seems bend the stick too far the other way and fly in the face of the many ways humans have modified their environment and adapted to those modifications and fairly often learned with only a few examples. And sure, plausibly a fair portion of those quick adaptation involve reworking basic facilities like vision. The whole thing is consistent with the neocortex sitting on top of the rest of brain and serving as a means to rewire it.

> A human can learn something through a fairly small amount of specific knowledge via written, verbal, etc instruction.

This demands the mastery of language which is a very high bar. Most animals haven't achieved this, so I think if you expect to get a computer to learn this way before being able to reason as well before you consider it having intelligence, you would also have consider most animals not having intelligence as well.

> This demands the mastery of language which is a very high bar. Most animals haven't achieved this

Then avoid language as medium. Many animals can learn from being shown an action once and apply it to themselves.

But it's not transferable. If you teach the AI to play checkers, you don't use that knowledge when trying to teach it chess.