It's weird how I drifted away from this article only after few paragraphs as if it was AI slop.
>They are, however, very good at pretending to reason. “We can ask [them], ‘How did you come to that conclusion?’ and [they] just bullshit a whole chain of thought that, if you press on it, will collapse into nonsense very quickly.
Could apply to both. You can see a contrast with the thinking of Turing, a mathematician:
>Her view collides with the 1950 paper by British mathematician and computer pioneer Alan Turing, “Computing machinery and Intelligence,” often regarded as the conceptual foundation of AI. Turing asked the question: “Can machines think?”—only to replace it with what he considered to be a better question, which was whether we might develop machines that could give responses to questions we’d be unable to distinguish from those of humans.
You can see a mathematician trained in rigor and proof recognises that "can they think" turns into waffle about how you define think and replaces it with a less subjective test.
So, if this text reads like a slop, consider perhaps it may be just personal promotion. I mean, "AI ethicist" with a book, denouncing experts for saying reasonable and arguably obvious things, because they violate some nebulous and incoherent "humanistic"[0] values? Nah, I think someone wants to sell us some more books.
--
[0] - Scare quotes because I actually like humanistic stuff; I just don't like stuff that doesn't make sense.
It is like people consider Wikipedia unreliable because anyone can edit it, but multiple studies have shown that Wikipedia tends to be more accurate than sources such as encyclopedias.
This is not people being wary of unreliable material, but being wary when they can perceive the unreliability clearly.
I don't know if evaluating reliability by the lack of awareness of unreliability is a recognised fallacy, but I can't see any reason why it wouldn't be.