Natural language benchmarks don’t measure AI models’ general knowledge well

Thanks for the response.

> [BART] is essentially acting as a QA-pair memorizer over the training data, and at test time, it just matches the question onto those seen at training time.

Not super-surprising.

> The T5-11B+SSM model we evaluate in the paper uses a special "Salient span masking" pretraining objective that does this to some extent (only mask words at pretraining time that are likely to be "answers" to factual questions), so in essence the pretraining task becomes pretty standard cloze-question answering

This seems an obvious approach for cloze-type questions, but it seems non-obvious how to extent beyond this.

Are you aware of any work probing the differences in the representation using this style of masking vs a more normal language model objective? It would seem to me that this is the key to significant progress here (and of course one would speculate that a representation that works well for this would also work well for all kinds of KB-related tasks).

Thinking about this for a few minutes things like masking names, colors and numbers (the things that neural representations often confuse) and then asking questions based on them might be interesting. I wonder if bAbI could be extended for this?

Your comment is interesting but seems to conflate two separate though related areas. The need for a body arises from the embodied cognition school of AI which suggests that intelligence is fundamentally embodied, hence the need for a robot equivalent to a body for truly understanding language.

However this does not necessarily have to be related to causality, and counterfactual statements about a causal model. The math behind counterfactuals and causality is actually well understood now (see any of Pearl's books). It does not actually require that a system be embodied, just that the system have some suitable (and correct) causal model of the world.

It would of course be amazing to have both in one system, but that is not required. An AI system that understood causality and language could be bootstrapped from causal models supplied by humans - or even other AIs :)

I disagree. There are other ways to observe and learn cause and effect than by physical interaction. We learn new causes and effect of many things each day by reading or watching. There are also most likely examples of humans who never had control over their physical world (through disability or whatnot) that came to understand cause and effect.

I agree that causal inference certainly requires ability to interact with the environment. In do-calculus this follows basically from the definition of the "do" operator.

As other people already mentioned, the "do" operator does not need to be related to the physical human scale environment. Causal inference could be useful also in interacting with the internet or other virtual environments like games.

Additionally, even humans do not have intuitive understanding of physical reality outside of our evolutionary environment. Nobody is able to intuitively understand quantum mechanics or general theory of relativity. Our intuitions of causality can beak even in relatively mundane environment like low Earth orbit. For example, you can fire thrusters to push you towards an object and still miss that object. There are actual missions in LEO that failed do to this type of mistakes of elite test pilots employed by NASA.

Of course, you can use mathematical formalism to reason about unfamiliar environments, but no matter how much time you spend learning about multidimensional spaces you will not be able to imagine 4D space or a quantum wave. But nobody is claiming that we do not understand language like "spin of an electron" because we have never experienced what a spin of an electron is.

Now to get to my original point. My original point was that causal inference is not a basic building block of intelligence. Formalism like do-calculus or logic are too brittle to be a building block of intelligence. You need something that is robust to noise. Something that can consume a tensor of pixels and process it. Once you have a system that is able to deal with this type of inputs then you can strive to do something like do-calculus or logic on top of it. But my argument goes further. My guess is that ability to do formal reasoning will emerge from the need to carry out complex tasks. Nobody will intentionally program models to do logic or do-calculus. Ability to do it will arise in models as a combination of the necessity of solving complex tasks and clever training techniques as it did trough evolutionary process in humans.

PS.

> So no matter how much you are able to reproduce these patterns, they contain no content. The content is in the reader.

To me this type of assertions do not contain any content.

"The content is in the reader." I don't understand that at all. Maybe I'm just a Chinese room :) .

Yes. Specifically the way human brain interacts with environment, the same way any AGI will need to sense and react to an environment, the more real the better.

An interesting example would be if the NN learns something from a book, also fed it with the all the critique on that book ( the environment reaction ). So it can possibly learn from a human what is good or bad in that book, etc.

Then both AI and humans can speak the same language ( on terms.of "feeling"/sensing ) and eventually learn the proper context, so less parameters are needed.

Do we really understand? What's the mechanism for that?

My thinking is that we humans have some ability to do things by rote until we have a good statistical understanding, at which point we use the statistical understanding mostly, and fall back to rote where we have to be careful. But, mostly, we're stats machines too.

No, I said the best way to learn to predict human text is to capture human knowledge. I did not say how best to capture human knowledge. Obviously doing gradient descent over a huge text corpus is massively inefficient for a human. Whether its the best way for a machine to learn it is another matter. After all, the amount of data that humans experience through their senses during formative years, as well as the "computational work" that went into evolving human brain architectures surely dwarfs GPT-3's training regime.

> Perhaps the required ANN size to approximate the AI / AGI function is so huge that it's beyond the number of particles on Earth

Perhaps there's a teapot orbiting Mars. It's unlikely tho. I think a better hypothesis would be that to model a brain (if we knew how) one would need the same order of magnitude of neurons and connections. This is completely unachievable now, and even if it were technically possible nobody has a faintest clue about how to train such a thing. We do not understand _many_ things about how the brain learns (no backprop!), how it retains and retrieves memories, etc, etc. The best studied system is vision, and computer vision does actually work pretty well by now, achieving superhuman results on some tasks. Thus far the most complex brain we've been able to accurately and completely model is not even a brain at all - it's just a handful of neurons of a worm. I took an online course in neuroscience a few years ago hoping to gain some insight for my ANN work. The situation with our understanding of even primitive brains of i.e. rodents is, well, primitive. We also don't really have math to accurately model more than a handful of real, biological spiking neurons - PDEs get way, way too complex for anybody to handle. But one could argue we don't have to: ANNs effectively operate in the frequency domain instead, where magnitude of activation approximates spike train frequency, and timing effects are ignored.

Don't get me wrong, I'm pretty sure if we do achieve real AGI, it likely won't be with anything even remotely like the current tech and science stack. Energy efficiency and compute density is just not sufficient. But I'm also pretty confident that even with the current stack we will be able to, eventually, approximate intelligence closely enough to suit many practical uses that seem completely out of reach today, including even some limited degree of logical inference, and dare I say, cognition.

If only people (including myself) weren't so chickenshit and actually seriously worked on (and funded) this stuff, that could even happen in my lifetime. As things stand right now, people are afraid to touch anything AGI related due mostly to fear of ridicule.

In my comment above I am merely referring to the conjecture that _if_ we had ANNs that are large enough, and _if_ we knew how to structure and train them, there doesn't seem to be anything that would _fundamentally_ prevent them from supporting AGI. Those are very, very massive "ifs" indeed, without even a faintest glimmer of light at the end of the tunnel.

Sorry, maybe it is a wrong analogy. What I want to emphasize is that there is no calculating time in nature. At least, we could not feel "lag" in our "Nature Server".

For the part of chips, did you mean that it runs the "for loop" as parallel in each micro-chip (or gate or whatever tiny structure it is)?

If a text corpus correlates with human knowledge (your "presupposes a certain model of the world"), then it necessarily encodes the portions of the knowledge it correlates with. That is, correlations drive the search of model parameters that explain the correlations. The stronger or more numerous the correlations, the more accurate a model can be computed. But the search space informed by this network of correlations is much more restricted than a naive brute force search, thus the correlations provide information about, i.e. encodes, the correct model.

Very good points! I would add that we don't yet have a complete model of the computational properties of even a non-neural cell, even though we know that all cells exhibit simplistic perception and decision-making.

This is not to say that we should expect some non-turing weirdness or anything 'mystical' like that, but that we just haven't taken the time to analyze the exact mechanisms and to model cells at this level well enough to be able to say that they are similar to any particular computational structure. Thus, to claim that the brain in its entirety is similar to any particular computational structure such as an NN is either wrong or trivial - the only level at which 'similarity' can be claimed is that they are both useful in computation, and likely turing complete. Anything more specific is fanciful thinking.

It's amazing just how many people believe that neural networks have a close resemblance to the human brain.

Causal. Imagine a video. Each frame is plotted in space, so instead of a sequence of 2d images, you have a cube. The z axis of the cube shows each frame of the video in turn. This is an equivalent representation to the representation that we are used to. We have just switched the time dimension for a physical dimension. What you are reverently calling causality, is only correlation along the z spacial dimension. In the same way an image can be compressed due to the similarities across it's 2 dimensions, so can a 3d representation of video be compressed along all 3 of it's axis. An appreciation of causality is only induction and induction is only a correlation, if a then probably b. Finding correlations in one of the dimensions you are more familiar with, like the pixels of an image is no different to finding it in time. Within language the elements of time are encoded, just as they are within the video represented as a cube. "The boy threw the ball, the ball landed and then rolled." Just because the time dimension is represented to the neural network within the input parameters of a single iteration of the neural network, does not make it any less able to understand correlations across time.