The first thing to determine is what your culture is, and then define what it means for someone to fit or not fit that culture. It's not whether you like the person, which is a mistake that a lot of companies make. Write it down - your culture isn't something that you should be ashamed of. Is your culture really "aggressive people who speak English well"? Would you put this on your website? Once you've defined your culture, filtering against it becomes much more clear I think.
Some core cultural values that I think all companies should share are honesty, integrity and accountability. Is the person willing to admit times that they made mistakes? Do they take ownership of these mistakes? Do they allocate credit to others when talking about their accomplishments (important for everyone, doubly so for managers)? Do they complain about former companies and coworkers? Do you feel that they blame others frequently? Are they upfront about their work history and experience? Is their resume honest - if you ask them about a project they worked on, do you find out that it's not really what they wrote? These are all cultural questions, and grounds to accept or reject a candidate on lack of culture fit for. These things should be explicitly important, and should be actively interviewed for.
Other grounds could be development philosophy or work culture. These are some examples:
- Perhaps you are in a shop that has a heavy focus on extensive unit and regression testing, and you meet an otherwise strong candidate who thinks that testing is important, but largely overrated.
- Or perhaps you're an established team, working on an established codebase, and you find a developer who largely prefers to rewrite software over maintaining it.
- Or perhaps you are in a team which has periods of high stress due to the nature of your business or organization, and you meet someone who doesn't work well under pressure.
- Perhaps the role requires someone willing to stand their ground, and you meet someone who seems like a pushover. Or perhaps the team already has some strong personalities, and you don't want to add another.
- Or you could be a company that values work-life balance, and meet someone who thrives on high pressure jobs.
- You might be a hectic new startup, and interview someone who prefers to have a clear career path, well defined tasks and meticulous tracking of issues and work (although seriously consider such a person, as they might just organize your lives)
- Or you have the reverse - someone who sees issue tracking as needless bureaucracy, in an organization that has a a fair amount of it.
However the danger is, unless it is defined, it becomes arbitrary. Realize that you are racist, you are sexist, you are ageist, and you are biased in a million unfair ways that you don't even know. Everyone is. And so having an arbitrary and undefined filter such as "culture fit" runs the risk of exposing those biases.