Imagine a bank on Ethereum that accepts deposits and allows withdrawals (first example here: https://learnxinyminutes.com/docs/solidity/ )
Anyone can deposit and withdraw, and no central party has the right to control the funds, if the contract is designed that way. As such, there is no custodial risk that the funds are stolen - and perhaps even no need to regulate the bank itself. (obviously, someone can steal if they get your private key, just not through the bank itself)
Lots of other examples. Bitcoin lets us set a credible limit of 21 MM, but in the real world you'd have to trust a central bank's word for the same.
Or the Lightning network allows peer to peer payments through counterparties. But none of these counterparties are able to steal the money midway through. This is akin to having 20 random people help you move a payment across the world, without any of them being able to steal the money.
Finance is full of these examples where we have to trust other parties, and crypto may have a huge impact there. Also, by reducing regulations and capital requirements, may make the space more ripe for startups to attack finance.