But actually doing so is a fantasy. Building bias-free selection processes is really, really hard. I don't think anyone actually knows how. (At least not useful ones. You can trivially eliminate bias by never hiring anyone, for example.)
It looks to me like this is the fundamental agreement between the two sides here. Both sides want to hire the people who will be best at the job, full stop. People who think James Damore is correct believe that the typical hiring process, or at least some achievable tech hiring process, already achieves this, and any modification is necessarily selecting worse candidates. People on the other side, such as myself, believe that the hiring process selects for far more than just merit, and diversity programs are an attempt to fix that.
So, yes, let's pick the best of the best. How?