Edit: What just got to my mind is that we will never gain more bitcoins then those 21 million, but we surely will lose them (people lose they keys, die, whatever). Will the price go up to "infinity" or will it suddenly drop to 0?
This introduces a very unstable system. If the price of bitcoin doesn't (double + increase with inflation), as the block reward shrinks, the rewards will become useless.
1: https://bitcoin.stackexchange.com/questions/10486/when-will-...
I happened to be paying attention that day, and there was a brief period where the network was small because everyone's miners couldn't participate until they were updated. My little single-GPU miner was suddenly rather effective because I had it back up immediately while the difficulty was very low. These days I don't bother turning it on, but when the scheduled forks occur there's probably still some opportunity for the little guys.