Hiring process was always hard for me to predict the future cultural fit. People are basically "selling themselves" during the hiring process, willingly or not. Their behavior is not "as usual" during that time. I came upon just a handful of people who were the same during the hiring process and later on.
So, although I do pay attention to potential cultural fit (and get rid of the extremes), it is not mandatory. I try to understand how the person think regarding the position I'm hiring, how they do the work; to see if what they need in order to do their job can be provided by others and if their delivery satisfies the organisation they'll be joining. And that's it.
What is understood as the cultural fit shows up along the way and can be altered, like communication with the team members or clients/customers, commenting the code, attitude towards making mistakes, etc. If a person does valuable work, these are the things worth spending time about - it will pay off during the first crisis of whatever type.
What is also more important than culture to me is - trust. This is something that new hires need to work on themselves for the first couple of months, but they are aware of it since the day one. And once they do build trust by their team and the larger organization, the "culture fit" is natural.
So, tldr: pay attention more if the person will bring value to your organization. Recognition of that value would help fit better those who don't fit immediately. Before culture - build trust.