I don't particularly object to the idea that we hold some of our intelligence in our environment. I think it's true, and actually forms the basis for my objection. What I object to is that we should naively allow that environment to be changed from one which offers manifold possibilities for a full range of embodied engagement (the natural world and man-made things which relate to our bodies in a way which does not significantly restrict our possibilities, e.g. clothes or hand tools) into one which encourages superficial stimulus-response behavior and instant gratification according to pre-fabricated programs.
It sounded to me like Don Norman had already accepted that idea to the point that he didn't even realize it was a choice. He worries about a future of greedy carriers controlling our phone-brains. I worry about the fact that our minds could become so beholden to devices that, regardless of whether those devices are open or privately controlled, we diminish the depth of our experience and our tendencies towards reflective thought.
If drugs were available that increased our abilities but made us less thoughtful, would you accept that without deliberation? Would it really be better if those drugs were easy to make at home or handed out for free rather than sold in stores? This is already happening with Ritalin and "smart drugs". They improve our ability to carry out mental work but diminish our ability to reflect on what we are doing.
"Smart" drugs and phones are only so in terms of instrumental reason, not in terms of creativity or contemplation of ends.