I have found "culture fit" to often mean problems with adjusting to the process and management structure of an organization.
I have worked with folks from corporate environments who never adjusted to having to find their own work and set their own goals in the startup world. And I have worked (briefly) at companies where I liked the people and the work but could not be productive or happy with the kind of micro-management I was expected to put up with.
That's why it's valuable to interview for "culture" (both when hiring and being hired). You need to talk out loud about expectations of how we work together and how we decide what to do. Have this conversation enough times, both in interviews and just chatting with peers, and you'll see things you agree with, things you disagree with but don't mind, and things that are absolute warning signs of toxic environments.