There is an argument that block propagation and orphan rates acts as natural barrier to the blocksize.

With a very large blocksize limit, miners who produce blocks which are very big, run the risk of a smaller block from a competing miner winning the race, as it is faster to validate and propagate over the network.

This limiting factor should mean the blocksize converges to a size that the network can comfortably handle.