I tend to think that lots of solutions could come from topics like those discussed in this book, with a lot of further development: http://probmods.org/
The conditions for causal inference being possible are pretty clear and have to do with the intentional modification of the local environment.
The intention to achieve some new environmental state, and your action to bring it about, is a dynamical activity that enables "deep" model building.
Causal inference is not going to be some "module" of the brain... it requires a body. When you place your hand on a hot surface, once, you immediately understand that it is hot. It does not require "induction" (as hume supposed). That is because our body identifies causes.
Its therefore pretty trivial to observe no NLP system understands language, or even can understand language, because it lacks this capacity to acquire language semantics via participation in environmental exploration. It has no body.
ie., you need to have experienced "on top", "green", etc. to know what "green leaves grow on top of trees" means. There is no meaning in the frequency co-incidence of symbols in text.
So no matter how much you are able to reproduce these patterns, they contain no content. The content is in the reader.
However this does not necessarily have to be related to causality, and counterfactual statements about a causal model. The math behind counterfactuals and causality is actually well understood now (see any of Pearl's books). It does not actually require that a system be embodied, just that the system have some suitable (and correct) causal model of the world.
It would of course be amazing to have both in one system, but that is not required. An AI system that understood causality and language could be bootstrapped from causal models supplied by humans - or even other AIs :)
Causal analysis can be performed, via Pearl, on datasets collected for causal analysis.
You still need some mechanism to collect the data, ie., the scientist. This requires solving the "relevance" (/framing) problem -- which, in my view, cannot be solved under a congitivist (/computational) theory of mind.
"Data" which is relevant to a causal hypothesis isn't selected via inference, the body "selects" it.
eg., when my hand is on a hot surface, it's temperature isnt "chosen as the relevant casual variable".
The body is the primary solution to the relevance problem. So you can't just "shove in causal math" into a computational system and expect it to grasp anything.
I also don't think bootstrapping will take you very far: causal models of, eg., dogs are very deep. ie., we understand their 2d, 3d, skeletal, behavioural, color, sound etc. "dimensions".
To say, "the dog was well behaved" requires an extraordinarily deep model of "dog".
The only way i see this being built is via play, ie., via hypothetical interaction with an environment -- as we do -- with bodies capable of discerning relevance.
There's a lot of past and current work on causal inference on observational data too.
As other people already mentioned, the "do" operator does not need to be related to the physical human scale environment. Causal inference could be useful also in interacting with the internet or other virtual environments like games.
Additionally, even humans do not have intuitive understanding of physical reality outside of our evolutionary environment. Nobody is able to intuitively understand quantum mechanics or general theory of relativity. Our intuitions of causality can beak even in relatively mundane environment like low Earth orbit. For example, you can fire thrusters to push you towards an object and still miss that object. There are actual missions in LEO that failed do to this type of mistakes of elite test pilots employed by NASA.
Of course, you can use mathematical formalism to reason about unfamiliar environments, but no matter how much time you spend learning about multidimensional spaces you will not be able to imagine 4D space or a quantum wave. But nobody is claiming that we do not understand language like "spin of an electron" because we have never experienced what a spin of an electron is.
Now to get to my original point. My original point was that causal inference is not a basic building block of intelligence. Formalism like do-calculus or logic are too brittle to be a building block of intelligence. You need something that is robust to noise. Something that can consume a tensor of pixels and process it. Once you have a system that is able to deal with this type of inputs then you can strive to do something like do-calculus or logic on top of it. But my argument goes further. My guess is that ability to do formal reasoning will emerge from the need to carry out complex tasks. Nobody will intentionally program models to do logic or do-calculus. Ability to do it will arise in models as a combination of the necessity of solving complex tasks and clever training techniques as it did trough evolutionary process in humans.
PS.
> So no matter how much you are able to reproduce these patterns, they contain no content. The content is in the reader.
To me this type of assertions do not contain any content.
"The content is in the reader." I don't understand that at all. Maybe I'm just a Chinese room :) .
If I say, "No! The glass is under the desk" you can immediately: find the glass, reason about why you havent previously found it, ask for the glass to be filled, ask for the glass to be handed to you, report on whether you like where the glass is..
You could say, "Why is the glass on the floor!?" angrily.
And by doing so communicate an expectation about where glasses ordinarily are, express a desire-frustration, express a confusion over the intentions of others..
The information contained in "No! The glass is under the desk" is VAST.
It is vast because our understanding of the world is vast. Not because the sentence as a lot of letters. Not because the words in it occur in a certain frequency. Nothing about the sentence itself is vast. It is profoundly shallow.
Intelligence is this vastness. Intelligence is moving effortlessly from "why" to "how" to "when" to "what" across domains, across hypothetical/counterfactual scenarious, across intentions/expectations.
This vastness is laden in animal minds, even, essentially constitutive of animal minds. And it cannot be found in "data".
It is grown by the complex biophysical process of learning, which requires a deep (& playful) interaction with the environment. The environment grows your capacities as you play with it.
“I just saw a huge fish, but then I realized I was out of worms. I should’ve bought more, but I spent $20 on gas instead.”
Here the causal order goes from last to first (minus the first clause)