Computers are great at brain augmentation devices - but if you spend all your time building the device, then you'll get a poor outcome and have wasted (most of) your life. That's the practical reason we want both mass produced hardware and transparent (and hence, controllable) hardware state. A world of augmented minds is much more problematic when a central authority can and does monitor everyone's mind, not the least of which because in such a state revolution would be trivial to eliminate even in it's earliest forms. And I believe all thinking humans have learned by now to distrust any system that can perpetuate itself so perfectly, and so indefinitely.
Fair. Although you could imagine a world where things like the original .Net trusted computing initiative succeeded in full force, and you could not purchase general purpose PC that let you execute any unmanaged code. Just look at the phone market, you can execute C code technically sure - but in unprivileged user space - with hardware you "own". (yes I know about rooting, but this is rhetoric about a hypothetical)