In my mind there are really three dimensions they can differentiate on: cost, speed, and quality. Cost is hard because they’re already losing money. Speed is hard because differentiation would require better hardware (more capex).
For many tasks, perhaps even a majority right now, quality of free models is approaching good enough.
OpenAI could create models which are unambiguously more reliable than the competition, or ones which are able to answer questions no other model can. Neither of those has happened yet afaik.
The fourth dimension is likely to be the most powerful of the differentiators: specificity.
Think Cursor or Lovable, but tailored for other industries.
There's a weird thing where engineers tend to be highly paid, but people who employ engineers are hesitant to spend highly on tools to make their engineers more productive. Hence all Cursor's magic only gets its base price to ~50% of Intercom's entry-level fee for a tool for people who do customer support.
LLMs applied to high-value industries outside of tech are going to be a big differentiator. And the companies that build such solutions will not have the giant costs associated with building the next foundation model, or potentially operating any models at all.