why are transaction costs so high? I thought one of the selling points was that the transaction cost is negligible because miners main incentive comes from mining new bitcoins?
1) The mining difficulty automatically adjusts to keep the rate of new blocks down to about 1 every 10 minutes.
2) There has always been a 1 MB limit on how large a block can be.
For a long time, #2 didn't matter, because all blocks were much smaller than 1 MB. But about a year ago, transaction volumes finally rose enough that all blocks since then have been at the max. That means that rather than just taking up marginal bandwidth and disk space, transactions are directly competing with each other for the limited space in each block. That's caused transaction fees to skyrocket.
The question of whether the max block size should be raised has been extremely controversial in the bitcoin world, and the "bitcoin cash" fork was primarily focused on this question.
Bitcoin doesn't scale in its current form [0].
0. https://en.wikipedia.org/wiki/Bitcoin_scalability_problem
Maybe you're thinking of how when more people are mining, it becomes harder to mine. This is because there is a global fixed rate of Bitcoin creation. Every 10 minutes, 12.5 new bitcoins come into existence. Miners are effectively competing for the chance to be the one to receive those new bitcoins. If mining becomes unprofitable, then some miners will drop out, causing it to become more profitable for those remaining. Miners don't set prices or rules; price affects profitability which either causes more miners to mine or some to drop out.
If transaction volume (and fees) stayed constant, then as the minting rate goes down, there will be a lower reward for blocks, and some miners will drop out of mining because it's not profitable for them. Miners dropping out will lower the mining difficulty, so the remaining miners will get a bigger cut. The system finds an equilibrium. Mining profitability only affects the mining difficulty and miners themselves (and the amount of resources necessary for a 51% attack, but as long as that remains obscenely high then the specifics don't really matter in this discussion).
How is the bitcoin community rationalizing this seemingly inescapable mathematical failure?
And the fact that there's a limited number of transactions that can be processed in a single block, and competition between miners to validate transactions as quickly as possible, complicates things.