The "conversation" interface is exactly the same workflow as one used with google search: user states a query, page loads, user adapts query because they are not happy with the result, until they end up with a suitable result which makes them close the tab.
So they have made this amazing query-response system which is far superior to google due to the summarization of query results from the global web and the auto-translation to present them in the user's native language. This is the type of raw query-response capability which many software engineers are trying to use in their agentic coding sessions.
However, after achieving such innovation, the AI startups consciously choose to apply social media KPIs to their query-response startup, which incentivizes all the dark patterns we have seen in their user interface. They notice that a certain subset of users can be tricked into believing that the startup's query-response interface has human-like qualities such as a name and persona.
This user cohort shows amazing metrics in terms of time spent on app, so they adapt their user interface and their system prompts accordingly. The AI startup doesn't have to care if the reason for humans accepting the illusion of a simulated human interaction is due to social circumstances (lack of emotional intimacy) or an underlying psychological vulnerability that the startup is actively exploiting.
The AI startup only cares if their "simulated human interaction" product receives negative attention from normal people who are not part of the vulnerable cohort, e.g. the suicides or the parasocial romantic relationships with the chatbots.
It is exactly the same as in the gambling industry: There is a certain subset of users called "whales" who are the cash cows for casinos, but if you look at the actual humans who are labeled with this term one can see pathological gamblers, most of which are ruining their lives and families. Casinos do everything to prevent people from jumping from their roofs after they lost all their money.
If AI startups can use simulated human interactions to make vulnerable people act against their own interests in the same way as casinos and social media companies do, it will allow them to make shitloads of money.
But if you're actually a clever person then be honest to yourself and others about what you are working on, and why these human-like features are really added to the user interfaces of OpenAI or Anthropic or the other AI startups.
So this is my framing of the situation.
I don't think this kind of problem can be overlooked by the insiders, and we might see some internal rifts along these lines: Will our AI startup simulate a human interaction in order to exploit our vulnerable peers, or will our AI startup focus on delivering the best response to our user's queries?
Because now we have local models, which - assuming one has suitable hardware - provide 80% of the utility in terms of a query-response knowledge base.
As we are currently seeing, the AI startups with billion-dollar funding have very big economic incentives to focus on the "simulated human interaction" part of the equation, because their investors need returns.
The biggest strategic blunder I see at Google. Because if Google actually changes their excellent query-response user interface to a chat conversation which simulates a human interaction with persona, name, and voice, then they knowingly pivot to the same social media KPI driven business as OpenAI and Anthropic are struggling with.