The phrase "we want to hire passionate people" tends to re-enforce that prejudice to me. It frequently seems like marketing speak for "we want people who will write code for us outside business hours without reasonable compensation".
It's when you see "passionate" without any context that it becomes a turn off.