> We have something that can talk intelligently, but might be doing it mechanistically. ...how do we even tell in that case?

We don't need to and we shouldn't care. If repeated interactions with such a system demonstrate problem solving ability, long-term and short-term memory, context-aware information processing and capacity for self-improvement, we've come pretty far from a purely mechanistic statistical model.