There is a reason the ADA exists. Because before then, everyone assumed businesses would cater to disabled people because they wouldn't want to lose out on the business. But it turns out they are such a small market, it doesn't affect their bottom line.
How many markets will be too small to be worthwhile after prices increase? I would wager it would be more than the contrary
You're arguing that the free market will serve disabled people. That's been tried many times, and it never works out. That's why the government mandates serving disabled people.
And let's not even talk about the ethics of suggesting that disabled people get charged more just for being disabled, something they don't have control over.
To your point that they are more financially limited — that is the reason why disability benefits exist.
It assumes very simple cause-effect relationships by default. See "fallacy of the single cause".
It cannot be countered except by elaborating on the complexities of this particular example. Which are then promptly dismissed out of hand because they are "special cases".
And as a result, the person using this rationale never learns their lesson that, actually, the world is much more complicated than that, and this kind of simple 1-2-3 doesn't fit with basically anything except those processes which are determined primarily by averages.