Maybe, but I was shocked to find out that OpenAI had been training on my private chats. I was sending it entire files of my source code on the assumption that my private chats were just that, private. Now big swaths of my source code have ended up in their training data.

I opted out a week or two ago, but the past code is there. If a competitor asks it how to solve a problem that I already solved, the model will be that much better at solving it than it was before. I effectively just commodified part of my code base.

The same goes for Slack: The fact that Slack started training on private chats alarmed me. Conversations encode patterns of processes at a business. And Slack isn't exactly a small fly-by-night operation. (Edit: In their blog post, they at least say they aren't using the data to train LLMs that will be public, unlike OpenAI is doing by default with ChatGPT chats.)

These companies wouldn't release unmodified customer data into the wild, but the zeitgeist seems to be that trained models are different.

That said, I agree with your point about my startup's core value. I don't want this to become a distraction that impedes business. So it isn't a top priority by any means. But it's definitely on my radar now as a rainy-day project.

I also agree that a nonstandard experience could well lower the quality internally, but I might just consider that a cost of doing business until there's more regulation around this.