As someone who also rejected a company because their drinking culture seemed like a big part of their identity (and I don't drink), it depends. In this case, I could sense it was big enough of a contrast to who I am that it wasn't the type of space I wanted to work. There's enough fish in sea to find a better fit.