I've spent an inordinate amount of time banning fraudsters from my own cloud platform. The bots that companies write to do this are designed to fight off other bots. If you have an appeals process, the humans that control the malicious bots will show up to abuse the appeals process. One of my favorite cases was a user that signed up with hundreds of stolen credit card numbers (all from the same IP address, as they are apt to do), got banned, and then opened an issue in our open source repo complaining of trouble logging in. Yeah, you're having trouble because I banned you and your entire network of bots.

There is a balance, of course, but if you haven't seen how much automated abuse there is on the Internet, be careful of what you wish for. Even the tiniest of the tiny services suffer from massive amount of automated abuse. At that tiny scale, it was nearly impossible for me to keep up with the abuse without the help of automation, very broad bans, and deleting related accounts by walking a reputation graph (like Google is being criticized for doing here). At Google's scale, I don't think there are enough humans on Earth to deal with the abuse. As a result, there are going to be some innocent casualties.

I don't think it's an ethics thing, it's simply not possible to run a business without some sort of process like this. There are laws, rules, and processes that could cut all this down to levels that could be managed by humans, but the cure might be worse than the disease. (For example, it would be great if there was a 1:1 mapping between your national ID card and your IP address. All of these stolen-credit-card users could then be imprisoned. But, you know that that's a terrible thing, because it will also be used against anyone criticizing governments or large corporations.)

People who consider this an issue either wouldn't work for Google to begin with or end up butting heads with management and leave the company. The benefits of self-selection.

There is also a lot of internal corporate training and team-building designed to redefine "ethics" to make this sort of stuff acceptable.

It's probably something they're not really aware of. My company does something kinda similar. We're an aggregator of sorts. We have all sorts of quality and safety rules and we de-list things all the time for violations. As long as we aggregate 95% of what's out there, our users are happy, so we don't have massive incentive to spend lots of time manually working with rule violators, especially when a good chunk of them are actually scammers manipulating the system and harming our users.

I'd be willing to bet that there was no one developer or team who was given a task such as, "run our audit AI on accounts and permanently blacklist them forever if that's what the algorithm wants"

It was more likely years worth of minor changes that got it into this state. Teams adding different algorithmic checks for various things, where the output is just setting a flag on the account. And other teams adding account termination logic for certain flags or a certain number/combination of flags, not knowing exactly how the flags are set. Though maybe that would be the bad task. I'm just spitballing.

Nope. You can tell by how it is implemented.

Even if they do, it’s not their responsibility to fix those problems. It’s government’s responsibility to step in and fix bad behavior that market forces can’t fix (like when it’s more economically viable to destroy thousands of livelihoods per year than to invest in human review processes)

They select for sociopaths coding these things. If someone is able to be bothered by these actions, they won't last long or will quit voluntarily and someone more sociopathic will take their place. It's a problem higher up the chain of management.