It is happening in insurance right now and it is an unmitigated disaster that nobody wants to address.
There is real data used for insurance premiums and claims payouts but it's being swapped out for AI slop, and the sales folks are getting bonuses for selling hot garbage and the executives are getting bonuses for buying hot garbage.