Ask HN: P2P crypto exchange - possible?

1. There's a huge rise of bitcoin and more people getting into cryptocurrencies.

2. As for now the only(?) way to purchase coins is using an exchange.

3. to purchase your first coin you have to pay with fiat money.

4. Few exchanges accept fiat money and charge crazy fees for it.

My question is - can we skip the middle man and create p2p exchange? the main issue i have so far: - how to charge fiat money and where to store it (escrow account)?