Hacker News Reader

Comment by bufferoverflow | Hacker News Reader

AGI must be multimodal. If it can't understand images, video, sound, smells, tastes, it doesn't have a full understanding of the world.

View on HN

Why limit it to just human senses? Imagine an AI with all of the above... plus sensors for electromagnetic fields, non-visible light, non-audible sounds, hyper-sensitivity to air flow / pressure, to humidity... no idea how you would even being to train these things into multimodality, but so many "senses" would be emergent.

I didn't limit anything. Read what I wrote.

I agree, AIs should have all the possible sensors. We don't have much data for the non-human sensors in human environments though.

It absolutely tickles me to think AGI would be like "I think it's going to rain, I can smell the petrichor" or ask for its salsa to be free from cilantro because it tastes like soap.

"Hey Siri, what does this taste like to you?" is such an absolutely unhinged interaction

You can ask a blind person what they think it is like to see.

You can ask Claude or GPT-4o what they think it is like to see right now.

You did not get my point: humans have the same issue.

Are blind people not intelligent? Is it actually important to have a full understanding of the world? What about people that grew up in isolation? I think there are still some tribes in the Amazon that have had little or no contact with modern civilization. Are these people not intelligent?

There are some deep philosophical topics lurking here. But the bottom line is that you can obviously have intelligent conversations with blind people. Doing that with a person that is both deaf and blind is a bit challenging, for obvious reasons. But if they otherwise have a normal brain you might be able to learn to communicate with them in some way and they might be able to be observed doing things that are smart/intelligent. And some people that are deaf and blind actually manage to learn to write and speak. And there have been a few cases of people like that getting academic degrees. Clearly sight and hearing are not that essential to intelligence. Having some way to communicate via touch or something else is probably helpful for communicating and sharing information. But just a simple chat might be all that's needed for an AGI.

Blind people have a lot less understanding of the visual part of the world. For example, if you have a chemical reaction, the result of which relies on color, they can't tell you the result (without using some seeing tool like a camera).

I think AGI implies it: I can learn something through audio and apply it visually and the other wat around. I don’t think that’s some abstract human quirk. Isn’t that what enabled literacy? It seems kind of obvious intelligence is beneath the modality, agnostic about it.

Is a blind person less intelligent than a person capable of seeing? If so, by how much?

In my view, intelligence isn't about what senses you have available, but how intelligently you use the information you have available to you.

I think human/animal intelligence at its basis is spatial-temporal. That is we can model and reason about events in space through time.

Our senses I believe map to this spatial-temporal model. Blind people can reason about the world the same way as those who can see, because what were really doing is modeling space, and light, audio, touch etc are just ways of gaining information

That sounds really hard to test. If someone fails entire categories of question but does better at the ones they can do and focused on, is that a good result or a bad result?

When it comes to "understanding of the world", I'd say the average blind person has less. But the gaps in their understanding are generally not particularly important parts.

>When it comes to "understanding of the world", I'd say the average blind person has less. But the gaps in their understanding are generally not particularly important parts.

Is understanding of the world equivalent to intelligence though? In my view intelligence is about optimalising the mapping from the percept sequence to action. In other words, given a sequence of percepts, does it determine the utility maximizing action.

Imagine two chess bots on uneven footing. One plays regular chess with perfect knowledge of the board. The other plays fog of war chess—it only sees the pieces on squares it attacks. In this case, the former could play suboptimally and still win against the latter. The latter can have perfect information about probabilities of pieces on tiles and act perfectly utility maximising in response and still lose. My argument is that the latter is still more intelligent despite losing. There is a difference between action and intelligence.

Similarly a human doesn't become smarter or dumber by adding or removing senses. They may make smarter or dumber decisions, but that is purely attributable to the extra information available to them.

On the other hand I suspect there's a level of board game complexity where being blind (or not having a well-integrated image processor) makes a notable difference in how well you can track the pieces. You have the same information but you don't have the same systems for organizing and tracking that information. You're worse at using that information, which is effectively a drop in "intelligence".

But then how to we compare with dogs who has better smell sense, cats that have better motor skills, birds better orientation and navigation, other animals that better see at night?

There were many people in human history that made big achievement even though handicapped: Ludwig van Beethoven, Steven Hawking, John Nash - but yeah they haven't been born with disabilities so had their all childhood to train their brain.

I generally don't understand this obsession about needing AGI. If current LLMs can be extended to humanoid so just get motor modality and keeping current vision, audio, text ability IMHO they will excel in many fields like currently they excel in text, vision, audio than most humans.

If you gave a dog nose and sharper eyes to a human I think they'd have a measurable advantage. But brain is most important and those animals are not getting anywhere near a human.

I've never heard of cats having better motor skills, can you elaborate on that? They don't seem very good at fine movement.

Depends what you consider better motor skills, but I couldn't fall head first from a two story building, land on my feet and shrug it off. My cat likely could. I'd also struggle to hunt a healthy bird without tools to assist me.

Then again, my cat lacks opposable thumbs and would struggle to draw a line on a piece of paper with a pen.

If you were the size of a cat you'd be pretty well suited to survive that. The flipping reflex is cool but falling that distance is mostly not a motor skills problem.

A cat struggles to move their paw through the air in a smooth straight line.

cats are amazing hunters with great reaction. Most likely small dogs or different animals won't handle such drop from 2 story building so well. Check some 1st person (cat) video to see how they behave, they can analyze and make decision very fast - human reaction time when driving car is very big comparing to cat reaction.

So this is what I find as motor inteligence. If someone can process and think very fast we consider them intelligent and in the same way we should consider cat smart how fast and well they can plan escape from dogs chasing them or they hunting for prey. Imagine how difficult would be to make robot to do this all calculation about different jumps etc.

multimodality would be very useful, but on the other hand humans can't see infrared, and can't smell ~most things that other animals can

Does it need a 'full' understanding? smell and taste is useful for biological life... but we don't need that in our thinking machines, do we?

I agree AGI must be multimodal. I don't think that multimodal is 'set in place', nor must it be conveniently, human-centricly mapped from our senses.

I think a big part of intelligence is being able to correlate things. Well, the more modes, the more ability to correlate.

For example, smell is a component of ER triage. Some different problems smell differently.

And if I had a robot chef, but the chef couldn't actually taste... yeah, not sure I trust it very far as a chef.

Among other things, The Fine Article argues that the current approach of gluing together various models of different modalities is, in the end, going fail to reach AGI. Better track to try to build a single model which processes multiple modalities all at once.

Brains have very obvious mode-specialized chunks. That doesn’t mean that’s the only way to do it, but it’s an interesting fact.

I wonder if it will end up being a game-like loop where it processes everything that's come in since the last delta. Here's, you know, 4x200samples of audio, 2 frames of video, and here's all the mems sensor data during the delta, etc

Then you just work on getting the delta as small as possible, I assumed 5ms for audio, e.g.

The way we need AGI is it needs to match us, we are evolved to operate quite optimally with the constraints and given physics in this world

I'm out sure it needs to be connected to senses to be multimodal - it's not like blind people lack GI because of not seeing.

Ask a blind person if some color scheme looks good. Ask them to color-grade a film. Ask them if the laser level shows level.

Blind people, while definitely not lacking intelligence completely, have major gaps in their understanding of the world.

Must it understand emojis?

I don't understand most emojis.

But I guess I never claimed to possess NGI

Reply on news.ycombinator.com