Lots of people would be surprised. Because they aren't educated about how these things work.

I'd wager that project managers working on these teams also don't have the understanding (let alone intuition) to judge whether these risks are present, and so will continue fighting for it. The wager is based on the presumption that someone who did understand these risks would pivot or otherwise not allow this initiative to continue.

PMs fight so hard because from their position their job is at risk if the project takes a nosedive. But they don't know what they're advocating for a lot of the time.

I know quite a few folks like this in FAANG and adjacent spaces.

But machine learning! Neural nets! Artificial intelligence!

All words that have meaning that becomes diluted in public perception, and lead to things sort of becoming a "black box", where we don't really know how the models actually do what they do.

Go back on this very site to August posts and read the comments about how infallible apple's special sauce of anti-privacy is. "It's not a hash it's a perceptive hash so changing a few things won't fool it" even though there was rumors that you could false positive it all day with flesh colored cake frosting with raspberries on it.

Or did I miss the sarcasm tag?