Then just be honest and say it. "I left this organization because I felt uncomfortable there. If you want more detail I can provide it."
And yes, as the hiring manager, I may want to understand the cultural issues at your last company. Why?
Because I want to understand (a) what you value, (b) how you deal with conflict, (c) will you be able to help me improve our culture here and (d) what cultural issues you may or may not bring with you.
If I'm going to help you be successful here, I do want to understand your background. I may not need all that information before hiring, but I will get to it eventually because it's pertinent to helping you and the team succeed.