I think it depends a lot on the specifics, if the background noise is loud and persistent enough it makes it harder to understand the other person. Hard to tell from OP's description but I'm guessing it was actively impacting the conversation if they're complaining about it.