This strikes me as very similar to the debate around the Chinese Room.
Of course if you accept the required premise of the argument, you must accept that either, one, we don't live in a universe that is a pure system of deterministic rules, or, two, nothing in the universe can have true understanding.
The Chinese Room argument, scientific materialism, or the existence of true understanding—you can have at most two of those in a consistent view of the universe.
To your point though, the more interesting case is people who would disavow the Chinese Room argument, but then end up using reflecting its views while argue against the intelligence of this or that system.
It can be read in its entirety at the author's site: https://rifters.com/real/Blindsight.htm