The main thing that limits the size is latency in propagating large blocks. But most miners already have most of the transactions in each block. All you really need to send is a small data structure that lets the miner reconstruct the full block, from the transactions he already has. An invertible bloom lookup table makes that pretty easy: http://www.i-programmer.info/programming/theory/4641-the-inv...
Using this, a home bandwidth connection could handle about 45,000 tx/sec, which is more than Visa at peak: https://gist.github.com/gavinandresen/e20c3b5a1d4b97f79ac2
The other problem is blockchain storage. They're planning to prune old empty addresses, which should help a lot. A more radical solution is to store the full blocks only for the last couple months, and only a linked set of hashes back to the beginning. That might be a bigger change than Bitcoin can handle, but another currency could do it.
All full nodes still need to process everyone else's transactions, so bandwidth requirements are still hefty, just not squared. Even if someone's home connection can theoretically maintain 45k tx/s, how many of those generous full node maintainers out there will be excited about the prospect of their Bitcoin node using their entire internet connection? Storage requirements will also still be fairly intensive, even if nodes prune everything but the UTXO set (that needs to be stored in RAM--although that can change--it's a half gig).
Not to mention that proposal can create problems where miners withhold transactions from other miners so they can build secret chains that other miners can't build on until they have see the withheld transactions.
All that aside, your comment still ignores the reality of the situation. The Bitcoin network has been bleeding nodes for ages, and has settled around 6500-7000, probably a little more[1]. What that says to me is that even if Gavin's 20MB block proposal works for him on paper and on his VPS, the reality is that we can barely maintain a healthy node landscape with the current stresses of the network. Upping it to 20MB will likely cause a "shedding" and we will come to the new normal, so on and so forth, until we increase the block size so much that we end up with only a handful of nodes.
This is all assuming the network continues grows to those needs, which I found doubtful.
I don't see how those nefarious miners could succeed, unless they were powerful enough to carry out a 51% attack anyway. Otherwise their secret chain will soon fall behind the public chain.
They and a few other projects also hope to pull off a "proof-of-stake" system that mostly does away with mining. Done naively that can be attacked, but there appear to be ways to mitigate the problem. Initially, Ethereum is settling for a proof-of-work algorithm that's more resistant to specialized hardware, so big central miners can't get such an advantage.
I've recently read a proposal for such a scheme written by Justus Ranvier, you can find it here:
https://bitcoinism.liberty.me/2015/02/09/economic-fallacies-...
tl;dr Bitcoin peer-to-peer (P2P) network needs price discovery built directly into it through micropayment channels (https://bitcoinj.github.io/working-with-micropayments)
tl;dr: optimizations like pruning, combined with a much larger economy, could result in a far higher full node count under a large-block-size scenario than there is now.
https://blog.bitcoinfoundation.org/blocksize-economics/
In the future, the blockchain inital sync should be torrented by default.
So if, for example, average fees increase by 2.5x and price increased by 10x, then transaction/block only needs to increase by 10x in order to raise this metric by 250x. Not only this is possible, but this is likely. This metric has increased by 1,000,000x in the last few years, why would it suddenly plateau right now?