No you just decrease the faith you can put into each confirmation.
> You either have a centralized system that can cheaply tell you if something is legitimate, otherwise this is the only way to protect a decentralized system.
There are some interesting efforts to create systems that work in complement to the main block chain.
One complimentary idea is the lightening network which functions by creating channels that only transact of bitcoins that are currently locked out of the bitcoin system. This eliminates the potential for double spending these bitcoins in a tx with a third party. Once you eliminate the potential for double spending you can do just about whatever you want quickly and off chain. The system is set up to be able to be able to resolve all of the txs that happened off chain when the locked up bitcoins are unlocked.
Once many of these channels are established you can start to establish a connection. For instance if I have 12 bitcoins in my own wallet I might keep ~1 BTC locked up to a connection to coinbase. From there they may charge a fraction of what the on chain cost for any tx would be. It'd take a regular on chain fee to establish the connnection, but once it is up the fee and latency should drop dramatically.