You are also ignoring the fact that for most of bitcoin's existence, block sizes were soft limited by miners as appropriate. With soft limited or flexible block sizes, just because big blocks are used sometimes when transaction volume is high does not mean that we will see gigabyte block sizes every 10 minutes for all of time.
The root of the debate is that small blockers have conveniently chosen to ignore the reality that markets and market participants can regulate themselves.
In the small blocker's mind, keeping the power in the user's hands means making bitcoin so expensive that the only people who can use it directly are banks and governments that offer sidechain connections to other networks. They blather on about decentralization while the reality is they are the ones creating centralization.
It is a questionably convenient choice for small blockers because it forces centralization to things that aren't actually bitcoin which is the most logical desire of governments or other bodies who wish to subvert bitcoin.