Bitcoin simply doesn't scale very well because of the way transaction history need sto be stored. As long as the rate of blockchain growth can't be significantly reduced, transaction rate can't grow very much.
So small block proponents are likely to be right in that on-chain scaling is unrealistic, but if the market must be forced by developers to adhere to an artificial limit, I think that in itself implies Bitcoin is unlikely to be able to grow very far as the market is unable to self-regulate.