The problem is, we don't really know how consciousness works (I assume consciousness the the part the author takes issue with; most of our cognitive faculties in isolation are not that special). We don't even have a great definition of consciousness, or good tests for it, or know whether it is a linear spectrum, or if it emerges abruptly with the evolution of certain reasoning and attention faculties, and we don't know which animals have it and to what degree.

So when people say we shouldn't develop AI to think like that, it's basically saying we shouldn't try to understand how consciousness works. Because as soon as we do, I guarantee someone out there will attempt to make conscious AI.