Others want Bitcoin to become a settlement layer, and certain backers have said that $1000 tx fees would be great, since that means BTC won as a settlement layer. This kills the peer to peer aspect though, and end users would rely on offchain third parties to actually get transactions done. This group has fought to keep small blocks, despite recent congestion and increasing fees. The argument is that increasing blocksize can't solve all problems so why even start, and that increasing it even a bit would somehow prevent regular people from validating the chain.
Complicating matters a bit is that some miners have an optimised way of mining, exploiting Bitcoin block header layout to get an optimisation during hash calculations (AsicBoost). This is a 20% advantage but also encourages mining occasional empty blocks. AsicBoost is incompatible with many suggestions of changing the Bitcoin protocol, giving economic incentives to object to such things. This muddies the water.
On the other side, companies have invested a lot into becoming clearinghouses (Coinbase) or inventing ways to do offchain tx (Blockstream) casting doubt on their intentions of killing P2P and making BTC just for settlements.
It's not clear to me why an increase to 2 or 4MB wasn't done though. Perhaps to force the issue?