It works out even if some customers are able to eat a lot, because people on average have a certain limit. The limits of computers are much higher.
Pay-per-token is really the only way it can work. If some kind of fixed monthly price is desirable, then there should be a quota the user can assign, and then the agent could e.g. slow itself down by 50% when 50% of the quota is spent, another 50% at 75%, etc, to make it last longer..
As a side thought, I wonder how it could affect an agent's behavior if the information of this token usage/limit was brought to it..
I wonder if there could be something like that, maybe even a progressive rate limiting, where after a certain number of tokens or another metric of use, then the speed slows down a LOT.
Not saying that I would love that as a consumer, as I'd prefer this all-you-can eat, unlimited data plan, but I wonder if that would be a compromise that could work, as it seems to have worked OK with the telecom space.
edit: the nerd in me loves the irony of me making the above comment and then later seeing your username as flux :-)
If an hour of an excellent developer's time is worth $X, isn't that the upper bound of what the AI companies can charge? If hiring a person is better value than paying for an AI, then do that.
They can charge whatever they want, I think many people like to make business decisions based on relative predictability or at least be more aware that there's a risk. If they want it to be "some weeks you have lots of usage, some weeks less, and it depends on X factors, or even random factors" then people could make a more informed choice. I think now it's basically incredibly vague and that works while it's relatively predictable, and starts to fail when it's not, for those that wanted the implied predictability.
I'm not sure how businesses budget for llm APIs, as they seem wildly unpredictable to me and super expensive, but maybe I'm missing something about it.