This makes as much sense as designing without different font sizes, designing without borders, or designing without rounded corners so everything is rectangular.
Visual design has a vocabulary where everything can be a useful tool for differentiation, but which requires a visual balance between too much uniformity and too much variety.
Arbitrarily removing color as a tool and treating it as an "extra" to be added back in later makes as much sense as writing a program that organizes code and variables into separate files instead of separate classes, and then adds back classes later.
Certainly, sites should still be completely usable by the colorblind, but color is in no way an "extra". It is functional, and treating it merely as ornament instead will lead to a more limited, more likely worse design -- the same as if you limited yourself to a single font size and were unable to draw extra attention to headings over body text, for instance.
And if you want to focus solely on layout? That's what wireframing is for, and you ignore everything else, not just color.