Hacker News Reader

Comment by miohtama | Hacker News Reader

The originally hack was caused by the fact that Silk Road was running PHP on MySQL without transaction isolation. Many early crypto exchanges had similar withdrawal bugs as they were running on LAMP stacks - MySQL has been notoriosly famous for having lax transaction isolation. Sometimes you could overwithdraw just by hitting refresh fast enough in a web browser.

If you deal with money use PostgreSQL + SERIALIZABLE transaction isolation level to be sure.

More in PostgreSQL documentation https://www.postgresql.org/docs/current/transaction-iso.html

Also Zhong was 22 years old script kiddie when he hacked Silk Road. Any smart criminal would have left United States long time ago if you sit on the top of $3B stash.

View on HN

A lot of the early Bitcoin stuff was toys which became something more than toys faster than the people running it could transform the toy infrastructure at it's core. In most cases, they didn't have the experience necessary to make it something better. Mt Gox was just a reused domain and was origionally an exchange for Magic the Gathering!

Note that modern MySQL has SERIALIZABLE transaction isolation as well:

https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-i... https://mariadb.com/kb/en/mariadb-transactions-and-isolation...

...how modern are we talking? SERIALIZABLE existed over a decade ago. It's mentioned here in a 2009 post.[0]

[0] https://ronaldbradford.com/blog/understanding-mysql-innodb-t...

As someone who very briefly poked around Silk Road at the time (just created an account), and as someone who has used PHP with the LAMP stack occasionally over the years, I am completely unsurprised. Race conditions can be pretty tricky to prevent in PHP, since the state is contained within different threads that can’t trivially communicate with each other. You need to be proficient with global state mutation such as via Redis or SQL.

Well, the database is there to manage and defend the data if only properly instructed to do so (CHECK constraints, locking, serialization.)

True, but it’s still not trivial, unless you’ve thought of the race condition problems in advance. My guess is that instead of keeping all the withdrawal logic in a comprehensive DB transaction with good constraint checking, the code was broken up into multiple sequential PHP statements with transactions of simpler SQL (which is likely easier from a coding perspective) and with insufficient locking. This is the easier way to code it if you haven’t thought of or heard about race condition issues.

Even if you're not 100% sure in your DB code you can do defense in depth approach and have the code that does the money operations do some checking too.

Even executing queued operations with few seconds delay allow you to notice say a series of withdrawals putting account in negative

Yeah, and it’s Bitcoin withdrawals, nobody expects it to be anywhere near instant. I doubt the market place set its transaction fees very high either.

But then you have to create a queue that only runs 1 transaction per second. I guess another imperfect but simpler way is to create a batch file of transactions, execute hourly, but randomize or sort the sequence in a way to make sure related transactions aren’t executed together.

Yes, I'm sure that's what happened. The money withdrawal scenario is the classic example for locking in any database programming classroom but not everybody went through that or remembers it.

At the same time, this is like... data integrity 101. For future reference if you want to build a system that manipulates valuable information (such as monetary accounts), and you've never done anything like that before:

Please pick a database (relational or otherwise) with ACID transactions and replication, buy at least two different textbooks on that specific database, read both, do any exercises they suggest, and then decide whether you still want to build that system. If yes, then cool: don't hold me responsible if anything goes wrong, but you need to learn somehow.

Yes, it's possible to build systems like that on non-ACID-compliant databases, or without using the ACID functionality. It's much harder, though, and you really must understand what you're giving up in order to have a reasonable chance of success.

Probably the code had logic such as the following, which from an amateur perspective, seems to work. After all, the first check prevents the withdrawal if the account balance is too low, so what could go wrong?

  $balance = sql_query(“SELECT balance FROM accounts WHERE account_id = “ . $account_id);
  if ($withdrawal_amount < $balance) {
    withdraw_funds($account_id, $withdrawal_amount);
  }

The correct solution of course requires putting the checks in an ACID transaction along with the account balance updating. This requires more work upfront (SQL is harder to write than PHP) and an impatient amateur coder probably didn’t think of it. Also I’m guessing they didn’t properly index all the DB columns and this made transactions take longer which exacerbates race conditions.

IMO this is backwards advice. Traditional relational databases with ACID go to great lengths to create the illusion that there's a single, global state, and that's what leads to people doing things like check-and-set that seem to work. You're far better off using a non-ACID database, simply because incorrect approaches will fail more visibly and more often. Ultimately you do the same amount of work either way, but using a non-ACID database forces you to do it upfront rather than letting you silently mostly succeed with a fundamentally broken data model.

This is a really optimistic take...

Realistically your DB of choice will probably try and tackle ACID half-heartedly because it's what devs expect, and that will lull you into the same false sense of security.

Many "eventually consistent" DBs will still have some concept that uses the word "transaction" that works like one until it doesn't (see DynamoDB and multi-region gotchas)

Your comment is not true for high quality databases like PostgrSQL. Check-and-set just works on the correct isolation level. There is no isolation, your transaction will fail and be rolled back.

If you are new to the subject there are extensive reports available from Jepsten, one of the leading names in distributed system and ACID anomaly testing and research.

https://jepsen.io/

> Check-and-set just works on the correct isolation level.

Which is not the default one.

> There is no isolation, your transaction will fail and be rolled back.

How so? Isn't it more likely you'll silently corrupt your data?

Even if it does fail noisily, this is a condition you're unlikely to hit in testing, so you'll probably be dealing with it in production.

Can confirm. Did a double withdrawal (of about 10$ at the time) on an exchange years ago by blocking a POST request and firing two at the same time.

I alerted the exchange operator and he let me keep the 10$.

And failed to even depend on bitcoin’s double-spend protections. He deposited and was able to withdraw those deposits within seconds: before any block was confirmed on the blockchain.

It wasnt worth anywhere near billions when he stole it. It wouldn't have been worth that much until very recently

It was around 50k - 250k USD in 2012.

Can anyone explain how a serialization anomaly as described in PG docs can result in a double withdrawal?

Seems to me that REPEATABLE READ mode should be sufficient, but I'm not sure.

Sorry, you lost me at PHP on MySQL for financial transactions. That's hilarious.

PHP isn't the issue here. Repeatly posting to any page normally would result in no security issue. If the balance is checked before each withdraw each request would be processed. Better method is a request is written to a transaction table and processed in sequence.

they would have arrested him overseas too

Yeah, look no further than the example of Kim Dotcom.

Reply on news.ycombinator.com