That makes a lot of sense, knowing what (little) we know about how creativity and connectivity work in the brain. Memories interact with each other and with the outside world, and that just doesn't happen when stored externally.
I can definitely say I've been feeling the effects of my "hard" knowledge gradually being replaced by pointers and references in the last 15 years. And it's not just that horrible "smartphone dies -> IQ drops X points" effect, but also declining creativity - regardless of my smartphone's battery.
So yeah, I agree this is probably a stage in our evolution and will be a non-problem once we upload to the net, but what can we do until then?