The problem is that in your scenario, many many more humans would be harmed. While animal models are flawed, there are no cell culture tools that even come close to recapitulating the system wide effects you get in a whole organism. Without animal tests, all the drugs that hurt a mouse (and there are very many that fail in mouse tests for this reason) would instead do it to humans.
You can ban animal research, but if you do, know that virtually none of the medicines you take would exist. It really is that stark.