I don't think they're speaking too strongly. I think a lot of the time when we correctly infer causality without empirically interacting with the system, it's because we have built up significant categorical experience about more atomic systems we were able to interact with.

In my view, a lot of things that are noninteractively inferred are compositions of more fundamental things that required empirical experience. When you've had the causality of gravity thoroughly beaten into you at a young age, a lot of other things seem intuitive that would otherwise completely fall outside a framework for being unempirically learned.

Do you have a specific counterexample of causality you can infer without interaction or empirical experience of something related?

Caveats: I'm not a neurologist or psychologist, so this is mostly philosophical speculation on my part.