Doormats gets bullied. I have seen this dynamic multiple times already and was on receiving end of it multiple times.
Shrugging makes them do it again and again, then others join, and originaln person escalates. And you not just be insulted, but they wont take your ideas seriously, cause they will have you fixed as someone without respect.
You have to stand up for yourself.