I'm sure it's talked about, but I've never really found a straightforward explanation of how it should be done - is there even an agreed upon set of best practices? If so, why are these supposedly sophisticated exchanges not able to successfully apply them?
Ikeboy(sibling comment)'s explanation is as good as you'll get, in my opinion. And you can even skip steps 2 and 3, as we have done for two years on BitMEX.
It's our opinion that customer funds are simply too important to expose to this kind of risk, so we process withdrawals only once a day from airgapped multisig wallets. It's a bear, and it's slow - we developed a few internal tools to make the review process easier and separate signal from noise - but aside from some manual work we've never had a problem.
The #1 surprise, to me, was that customers don't mind the inconvenience. Sure, a few wonder why they can't simply get their funds immediately. But it's explained to them, they seem to appreciate the sensibility of managing this risk seriously. Compared to alternatives like ACH, at least it only takes one day and runs on weekends.
I can't imagine a nightmare worse than losing millions of dollars of other people's money. I hope for Bitfinex's sake that they are able to recover a portion and continue operating. We've spoken personally with much of the team and enjoy the relationship. They - and their customers - don't deserve this loss.
Waiting a day for a transfer now seems hilariously outdated now to me.
I understand there are other benefits to bitcoin but something seems inherently wrong if transfers have to be intentionally delayed so long to make the system more secure.
My understanding is that the inconvenience of ACH is by design - they are security delays. If you need to send money from one account to another instantly, you use a wire. If you have the amount of money in your account that is typical of people that need to send wires, they are often freely included with your higher-end checking account. Wires send the money instantly.
I commonly see wires hit within ~30 seconds of hitting the "confirm" button, and almost always within <=5 minutes, and that's between separate US banks.
2. Have a warm wallet on a computer with a firewall that only connects to 3, with 1% of your funds
3. Have a web-facing server that sends back commands to 2 when it needs to send money.
2 alerts someone when it needs more money, who then goes to the cold wallet and transfers some, creating the transaction offline and using a new USB key to transfer it to a computer with internet.
The last step can also require multiple people if desired.
If I wanted to make a trustworthy rate limit, I'd have the machine from (1) not airgapped, but connected by a serial (unidirectional) line to the machine from (2) along which it would periodically send signed transactions sending some of the money from "lukewarm" storage to hot storage, and sending change back to the very same "lukewarm" storage address. The machine from (1) can then implement a rate limit.
Edit: actually nvm, didn't get the part about unidirectional. I don't deal with networking but I assume that's possible physically and there's no way to reverse the flow?
Sometimes known as a Data Diode.
[0] https://www.reddit.com/r/Bitcoin/comments/4vtuxo/bitfinex_se...
Here is a report which uses the guideline to assess a company's posture after a hack: https://www.scribd.com/doc/309591980/ShapeShift-Postmortem (sorry about the scribd link, haven't been able to find pdf.)
Here is a narrative description of the hack that the above report is about: http://moneyandstate.com/looting-of-the-fox/
What good is that proof of control? Literally the next second it could be drained and no one would know until the next scheduled proof of control. I've never been sure what publishing proof of control is meant to protect against exactly.
There are more sophisticated schemes that allow you to check the amount in your account, and verify that your account was included in the total, without seeing the size of other accounts: https://iwilcox.me.uk/2014/proving-bitcoin-reserves .
Though in this case it looks like it was a hack on a third party provider, which is why relying on third parties for core functions is not a good idea. Why would Bitfinex use that provider instead of a multi-sig cold storage is anybody's idea.
Why bother? It's much more profitable to pretend to secure the wallet, and then take the money and run.