I think small blockers are concerned that short-term fixes like increasing the block size will cause long-term scaling solutions to never be developed, necessitating more and more short-term fixes.
Which to me is just stupid - like taking 5 min to increase the blocksize is going to take away significant development effort from more novel solutions. Meanwhile, tx fees are stupidly high and bitcoin can't be used as a currency for small-ish transactions.