The anti-block size increase camp says that increasing the block size is only a temporary fix, and that no blockchain can accommodate Visa levels of throughput.
They also posit that any increase in the block size will make running a full node more difficult, meaning Bitcoin will become more and more centralized and consolidated around the big mining pools that have the hardware to handle bigger block sizes.
However, many in the anti camp are pro segwit (segregated witness), a code change that does several things and is the first step to enabling off-chain networks like Lightning Network. These are basically pools of Bitcoins tracked by networks off the blockchain that would allow you to instantly exchange Bitcoins to anyone else in your network.
This is a big problem for miners who have poured lots of money into their rigs, because now the blockchain would be little more than a settlement network. There would be far more transactions occurring off the network, meaning less money for miners.
Again I'm no expert on this but this seems to be the crux of the debate. The Core developers seem to be OK with having the blockchain relegated to being a settlement network (similar to how Visa, your bank and merchants work together), but many people are opposed to this, partly because these off-chain networks would quickly form large central "hubs" which would obviate many of the benefits of Bitcoin (anonymity, privacy, security and decentralization).