Eric Brewer on Why Banks Are BASE Not ACID – Availability Is Revenue (2013) | Hacker News Reader

Eric Brewer on Why Banks Are BASE Not ACID – Availability Is Revenue (2013)

highscalability.com

102 pointsmpweiher41 comments

Interesting article, but it's really talking about ATM networks rather than the broader "banks". A bank's core systems could still hugely benefit from having an ACID compliant database even if any individual ATM node could still be allowed to operate independent for some amount of time to maximize availability.

Consistency is a feature whose importance is regularly downplayed by hawkers of NoSQL databases. I've seen firsthand what happens when a NoSQL database scales out and it's not pretty — a significant number of people end up spending a significant amount of their time trying to figure out how to maintain data integrity across failures in the system, even the kind of minor failures that are occurring every day. A lot of the time that involves going in and cleaning up data by hand. There is no general solution for this problem, and it'll be a persistent operational tax that dogs you as long as your NoSQL database stays in use.

No the article uses ATMs as an example but the general accounting principals apply regardless. Consider paper checks or IOUs. Humanity literally has thousands of years doing eventual financial consistency while ACID compliance has only existed for the last 50 or so.

I'm not saying ACID isn't good (or that NoSQL is a panacea) but we have long had exceedingly successful financial systems without ACID.

Define "successful financial systems"

Every bank that existed before 1960 starting with ancient Greek temple depositories all the way through the first "official" banks of 12th century Venice.

Historically banks were distributed - your account was at a particular branch, and transfers needed a remote protocol.

It is easy to laugh at Bitcoin but with Bitcoin you can send $15 from New Jersey to France immediately but the fee to do a bank transfer is many times that.

I wrote a check for $130,000 dollars that bounced once because the teller deposited the check I got for selling a house in somebody else's account. Mistakes happen, but the bank is able to figure it out and (hopefully) make it right - that is why the system is built to tolerate slop.

If you'd sent it to the wrong wallet, OTOH, you'd just be at the mercy of the wallet owner to return the BTC.

These are real problems, and by the time bitcoin proponents find adequate solutions for them, what they will have created is going to look an awful lot like the banking system.

> If you'd sent it to the wrong wallet, OTOH, you'd just be at the mercy of the wallet owner to return the BTC.

IIRC, a wallet is just a public key, so nearly any incorrect account number will be a public key for which no-one has created the private key — thus there would be no-one to return the money.

This really isn't a problem, I think, because I'd rather have to rely on me and my computing equipment not committing errors rather than have to rely on some random teller not making errors.

Your point makes no sense.

If some random teller makes a mistake the bank can just organise for the transaction to be reversed.

> If some random teller makes a mistake the bank can just organise for the transaction to be reversed.

It is not that easy to reverse an international wire transfer, especially if the funds have already been withdrawn.

Not easy, but possible. With bitcoin, not so much.

You've seriously never copy-pasted one character short of the full line?

You have a far greater estimation of your own infallibility than I have of my own. Or yours, for that matter.

If you really believe that, I know some spear-phishers that would love to make your acquaintance.

If I were to use BTC for core transactions, I would: 1. Make a system with far more checks than that, for instance force to provide a name and an account number and refuse the transaction if they do not match.

2. Have an insurance system for that kind of mistakes. And, obviously, this would mean added fees.

Odd, I transferred tens of thousands between the UK and Australia for not much more than those 15 dollars.

No, they likely just hid it in the forex spread. No bank is doing $20,000 retail cross-currency wire transfers for a $15 flat fee.

So what?

I transferred cash across the globe for peanuts. The underlying mechanism doesn't really matter to me.

No, you're just not aware of how much you paid.

For most of the big US banks an international wire costs about $45. That's to send $15 or $15,000.

So, did you try sending $15?

For a small amount like $15, I'd use PayPal.

For larger amounts, TransferWise. They take a 0.5% fee for the conversion, but the transfer is usually fast.

TransferWise costs 1% (at least for me, for USD -> CAD) and takes days.

It's incomparable to Bitcoin which takes 10 minutes and is practically free (including the BTC<->fiat conversions if you fiddle with limit orders).

It seems to be 0.5% for most European currencies, and from other countries I expect to have good banking systems (Japan, Hong Kong, Australia etc). Perhaps that reflects lower costs for them.

Although it can take days, I've usually found it takes at most a few hours. I transfer between GBP, EUR and DKK.

> It is easy to laugh at Bitcoin but with Bitcoin you can send $15 from New Jersey to France immediately but the fee to do a bank transfer is many times that.

Then use PayPal.

One of the few uses I have for PayPal is small value transfers and purchases.

This is a very selective view of banking. In financial reporting you have a lot of situations where you need a consistent point-in-time snapshot of your data or you might be in very big trouble.

It's true that clearing of payments and trades is usually a multi-step process that isn't done in one huge distributed transaction. But implementing each stage of that process without ACID transactions seems like a daunting task to me.

So I think Eric Brewer's remarks lack a bit of nuance and detail, but he's not entirely wrong of course.

Could you be specific. In what particular areas do you need a consistent point-in-time snapshot?

Having worked in finance myself, as part of a team that procured and replaced an international core banking system with a state-of-the-art, multi-hundred million dollar successor; we've never had a strong consistency requirement. Much weaker forms consistency sufficed for their needs and the requirements of regulators.

Also what big trouble do you mean? I do not know of any financial regulators that expect banks to have a totally consistent point-in-time view of their balance sheet. Furthermore the Basel Accords and similar banking supervision rules do not mandate such a requirement.

I be very much grateful if you could provide me with this information, as I can pass it to my former colleagues who will be quite surprised to discover this requirement.

A long time ago I was part of a team that was supposed to create a new core banking system. The project emerged from an earlier attempt to remodel the entire IT of a smallish European investment bank.

Some of my coworkers were still involved with that previous remodelling work, which was based on an extremely weird 1980s/90s RAD tool + database combination (I don't remember the name). That system was lacking proper ACID semantics and they were struggling very hard to create consistent reports.

The bankers kept throwing back their reports because of inconsistencies that resulted from an inability to get a point-in-time snapshot while the system was in operation. So they tried to create those reports during the batch windows available to them, which turned out to be just barely possible after many embarrassing failures.

This will probably leave you unsatisfied because I cannot tell you exactly what was in those reports or why selecting based on a date column wasn't enough to get a consistent view. Accounting is supposed to be append only after all. I can only tell you that they were smart people struggling with the lack of ACID transactions.

I seem to remember that some reports were for the banking regulator of that country. The bankers were extremely concerned about any delays, because they feared closer scrutiny by the regulator. That's one thing I was referring to when I said "very big trouble". The other one is that the software maker's contract with the bank was apparently in jeopardy.

That last paragraph implies you're less interested in finding out and more interested in insisting those requirements don't exist because you worked on a project that overlaps with the incredibly broad term "financial reporting". The term is so broad I'd almost say your anecdotal experience is more likely to be irrelevant than not.

A core banking system captures and stores all the transactions and business a bank performs. Without this a bank would not be able to function let alone produce a meaningful financial report.

In what way does that mean you'd be able to disprove the statement:

"In financial reporting you have a lot of situations where you need a consistent point-in-time snapshot of your data or you might be in very big trouble."

You're trying to sell your anecdotal experience rather than explain your point.

I find quite often in India that whem networks are down, ATMs go down displaying a network failure error message on screen. I've never heard of anyone being able to overdraw an account. Credit cards also need authorization both through PIN on terminal and bank authorization over the networks (which do fail from time to time). Over the Internet, two factor authentication is pretty much mandatory especially for credit cards.

Therefore this whole Banks don't value consistency argument feels very fishy to my experience. Is this a uniquely American thing like lack of chip and pin?

I don't know about India, but I know that in many western European countries overdraft fees such as those that are common in the US would be considered above the maximum legal interest rate, because of that the bank has no big incentive in letting you withdraw money you don't have. Consider also how much more widespread chip and pin cards are on this side of the pond, and you're starting to make me suspect that the wider adoption of chip and pin might have been helped by usury laws.

1234 days ago:

https://news.ycombinator.com/item?id=5642010

The conclusions of this article have far more implications for the future of blockchain technologies than for the future of databases.

Bitcoin and Ethereum and their ilk have, out of necessity, valued ACID qualities above all else- including availability and responsiveness. Does it make sense that completing a $1 transaction would require the same amount of resources as a $100,000 transaction? The traditional banking system has the flexibility to expend fewer resources on the $1 transaction, but Bitcoin does not- by design.

Interestingly, I think it is the inconvenience of the traditional banking system that gives it this flexibility. If financial transactions were something that could be done instantaneously, by a computer, without having to provide personal information to set up an account, then stealing money in 100,000 $1 transactions might be easier than stealing $100,000 in one big heist. Add a little inconvenience to each transaction and this problem goes away.

Unless blockchain transactions are extremely cheap, it will limit their competitiveness with the traditional banking system for this reason.

> Bitcoin and Ethereum and their ilk have, out of necessity, valued ACID qualities above all else

They definitely don't have ACID properties.

The blockchain gives you something like eventual consistency, but it's probabilistic.

You have atomicity within a single block, but given the acceptance of the block is only eventually consistent, I'm not sure it's meaningful.

You can get a benevolent miner friend to process your blockchain transaction for free

Gotta send them the signature and broadcast it directly to them

This is such an important article for people to read, especially people so bent on strong consistency. I've done talks on this for our database (eventually consistent, http://gun.js.org/ ), and people always ask about banking. Yes banks should use strongly consistent approaches, but they don't - user experience trumps perfect academics. And having Eric Brewer say this is fantastic evidence.

What do you do when your reservation app gives two people the same table?

What do you do when your auction app thinks there are two winning bidders?

It's not about "perfect academics" it's about risk management. How do you prevent or manage inconsistencies.

These are great questions and I hope I can convince you with some simple answers.

We cannot get this problem if we treat the table as its own thing. So for instance, rather than having a "reservation" that has an "assignedTable" column with a string of "2A", if instead we have a table that has a property of "7pm" with a value of "Mr & Mrs Smith" then this value cannot be a string munged with the other couple.

This easily applies to many other areas, like finance. Rather than having a "purchase" record that has a "debit" amount on it, make the software model the real world of currency. Instead you have a "dollar" record with a unique serial number, it has a "owner" property of "lotyrin". No matter how many times we concurrently write to this property, the end result must necessarily be 1 value.

This is the way humans have been thinking for thousands of years. And guess what, I sat down with a large airline and had to warn them "we're not Strongly Consistent" and they laughed at me saying "you realize we've been booking seat reservations before there was internet, before you were born, and before there was cheap telephony. Seat reservation has never been strongly consistent - we used to have hundreds of travel agents booking seats and it would take 2 weeks before we would hear about it."

Again, I want to emphasize this is exactly the reason that perfect academics doesn't always help - it is a solution to a problem in isolation, devoid of its real world context. I am in no way saying the academics aren't useful or interesting, but I am saying that even Eric Brewer himself recognizes where academics aren't helpful.

But to stress your point, let's take your example further. Rather than assuming the restaurant actually has a credit card authorization server that is the source of authority, let us pretend it is more of a bitcoin crowd hosted reservation service. Now despite our "Mr & Mrs Smith" value being straightforward, we wind up having a replication problem. One server has it as that value, and another peer-to-peer server has it as "Mr & Mrs Lionheart". What happens?

Well, under our database (like an Open Source Firebase, see the link in my original comment), this is why realtime updates and push notifications are so handy. Even if 2 couples reserve the same table on different peer servers at roughly the same time (such that when the peer servers checked availability there was no conflict), when the new update saves it gets pushed out to all the peers and runs through a conflict resolution algorithm locally. So all peers sync based off of push, not an outdated pull.

In fact, we made a great explainer video for this type of situation, see http://gun.js.org/explainers/conflict_resolution/lexical_sor... (might change the URL naming scheme in future, so careful of a broken link). But I'm going to again assume you aren't talking about "what practically happens" but are intentionally trying to push for the extreme edge case bugs. If so, let us add that the peers are offline while these updates happen (because yes, this is something we support, being offline-first) so therefore cannot sync with each other fast enough for the 2 couples to realize one of them got ousted.

We are now only left with two scenarios: A) Because when the app tries to save the data it gets an error that the push notification failed due to the network being down, we display a warning to the user that the "Reservation is pending, proceed at your own risk or try again later". Or B) Because we are being academic, we yes go ahead and make the reservation (unaware of the conflict). How do we address it? Well...

Finally, both the Lionhearts and Smiths show up at the same time. Initially they argue both asserting "I have the reservation" until a hostess steps in. The hostess apologizes and says "We are terribly sorry, this is our mistake and let us fix it for you." They then offer:

A) Would one of you like to sit at the open table over here, that somebody canceled last minute on?

B) If (A) is not available, then here are free drinks* for all of you, but who would like a 20% discount* off their next dinner? Next open table should be in about 45 minutes?

*Total loss is 75cents on sugar water and $10 of discount, but profit GAIN is $50 (because had the reservation app said "booked out" the 2nd couple would have never turned into a customer), leaving the restaurant at a net of +$39.25 at the end of the night.

Again, let us make it more extreme, both couples are uncompromising, saying "We cannot do that, we have a show at 9pm", then there are only 2 remaining situations:

A) You flip a coin and 1 couple leaves angry and never comes back to use the 20% discount because they are so angry the coin did not choose them. (Note, the restaurant still made a profit filling out the seat though) OR

B) You basically flip a coin and 1 couple leaves bummed, but enjoying their free drink and looking forward to coming back another time with a 20% discount. (Note, restaurant still makes a positive profit)

The moral of the story is that you want to create Win-Win options for everybody. Why? Because as you say, it is all about risk management and the probabilities that Murphy's Law is going to strike so many times (multi-continuously-offline-peers-that-do-not-error-with-above-average-unsatisfiable-angry-humans) such that you get an angry customer with a free drink and a coupon is extraordinarily unlikely. And if you have not already calculated for ever losing 75cents on sugar water plus a discount, then something is wrong with your risk management department - not with your algorithm.

With that, I'll leave the bidding app to your imagination. The trick question for finding simple solution is: What would humans do before the telephone?

I hope this was a pleasure to read, and I want to thank you for calling me out by asking the glaring question. If you have any concerns or counter arguments, please let me know. I would be happy to hear them. If you like what you see here though, check out the rest of our work on the Open Source database. Cheers!

While interesting, this is not satisfying as a general argument for non-ACID databases. Incrementing and decrementing are about the simplest possible operations, and even the article admitted that these are commutative, so the order doesn't matter. It is indeed much easier to work with operations that can be ordered arbitrarily, but I think that most real-world problems are sequential.

(2013)

It would be interesting to know if BASE at ATMs had anything to do with

http://www.theatlantic.com/international/archive/2016/05/jap...

Coordinated attack over two hours? A little DDOS + BASE perhaps?

I could only read a third of the article before starting to crinch too much. If this was on purpose and I missed the same conclusion I am presenting here, sorry.

The article sounds like "this of course is the smart way, make money first, worry about details later". But if you think about it this is very, very wrong. First, if any system will fail drastically at some point. You wait long enough and something unexpected happens. Second, if this consistency tolerance becomes widely known people will try, and at some point succeed, to cheat the system big time.

If any of these two scenarios happens, the whole system breaks. And then, what do you do? You can only get so many bail outs from the tax payers. The smart way, of course, is to not be too greedy and stick close to a relatively consistent scenario.

Reply on news.ycombinator.com