Blindsight by Peter Watts also discusses what can be intelligent but not conscious. In the current hypefest of LLMs it’s interesting to consider that they may be similar.
which then begs the the question, what is the magic ingredient, on top of use of language, that we have that bestows these qualities?
and also the observation that whatever this ingredient is, it must be very difficult to measure or prove which is maybe why we stuck with the crude, but easy to wield, "use-of-language" test for so long.
That said, your analogy may well be perfect, as they are learning to people-please and to simulate things they (hopefully) don't actually experience.
(Not that it changes your point, but isn't that Machiavellian rather than psychopathic?)
[0] one of many reasons why I disagree with Wittgenstein about:
> If there were a verb meaning 'to believe falsely', it would not have any significant first person, present indicative.
Just because it's logically correct, doesn't mean humans think like that.
If that's the function which we can now automate at scale, it's not the jobs the machines will ultimately take; it's the leadership.