Honestly, I don't understand why you wouldn't want the most accurate AI models available. The LLM is only as good as the data set it's trained on, and the more I read about LLM's and the advent of AI evolving from them, the more I'm starting to think if we don't jump both feet into the pool, then we'll never get to the promised land of:
"AI model, spin me up a T-shirt company that's scaled to 10mm users a month, and spin it down after 6mo. if sales don't increase by n% month-over-month"
or
"AI model, get me [A,B,C, ...n] groceries so I can throw a housewarming party on Friday. I can only accept the delivery Tuesday or Thursday. I don't care which store(s) those ingredients come from or how the internals are orchestrated."
What's the threat model here, specifically? What nefarious things would happen by using customer data? Most companies exist to make money, which honestly, is a pretty benign objective, all things considered.