Why do you think that? I cannot imagine anyone losing one second of sleep over it.
In this case, they can't even argue we're depriving creators of revenue they're owed. The only point of journals is peer review. Experts in the field that filter out the bad articles and publish the good ones. Peer review is the only thing that separates these glorified journals from a random website. Those experts don't see any of this money due to concerns over conflicts of interest. These reviewers could get together and create a blog and it would be better than these journals.
The way I see it, these journals have literally no reason to exist. It's unethical to pay them even a single cent for access to anything. The ethical and moral thing to do is to hasten their demise. It is our duty to society.