It's less about the company, and more about your reputation with those around you. Other people are probably fleeing the sinking ship too, but you want them to remember you as a person who at least tried to do the right thing. Walk out with no notice, and that'll be something they remember - and not in a good way - not something you want when they're about to hop into the hiring pool too.
I've worked in a toxic work environment in the past. I don't think any less of the people that I watched quit on short notice after continued mistreatment from management; in fact I respected them for not putting up with any more crap when the didn't need to. Assuming this environment is as toxic as described by OP, it's important to remember that many of your coworkers around you will empathize with your decision and respect you standing up to the manager that they too resent.