A meritocracy based on raw intelligence and skill (measured by standardized testing) is not exactly what people mean today when they discuss "promoting diversity". In fact, nowadays people typically criticize such practices as inhibiting diversity [1].
As for culture fit, it's a complicated thing. There are some culture issues that can be a real problem - for example, my company really can't deal with people who don't believe in "let data make the decision". Similarly, there are companies out there which highly value civility and team spirit over correctness - my "this is broken, here is why" style would not work for them.
A lot of culture fit is just "do I think these guys are cool" silliness, but don't let that overshadow the fact that culture matters.
[1] http://ashedryden.com/blog/the-ethics-of-unpaid-labor-and-th... http://flosspols.org/deliverables/D16HTML/FLOSSPOLS-D16-Gend...