Like, imagine GPS navigation wasn't widespread and there was a paid service that gave you 20 free trips. Eventually your normal navigation skills would atrophy and you'd be obliged to purchase.
Even if they explicitly and clearly state that they don't use your data for any purpose other than generating immediate responses, they can change this once the free Copilot gains traction and people become really addicted to it.
Like, "pay if you don't want us to use your data for training". Most people won't pay and will be happy to give away their data instead.
GPS is something that wouldn't be invented these days. Instead of the US military footing the bill, it would be some private company somewhere.
Real AI is definitely a rubicon.