Comment by _vvhw | Hacker News Reader

Comment by _vvhw | Hacker News Reader

Parent Full thread

"We need to keep track of our customers’ account balances. We need them to trust that their Wave balance is correct and Wave won’t lose the money."

"At Wave, though, we prefer to use boring technology, and a simple relational database like Postgres does an equally good job at this."

How does Wave ensure multi-region durability for Postgres, high availability, and strict serializability during failover, without risk of split-brain in the presence of network partitions?

I'm also interested in Wave's storage fault model with regards to Postgres? How are misdirected reads/writes, lost reads/writes, corruption in the middle of the WAL, and corruption at the end of the WAL handled? To ensure that account balances and transactions are never lost? As far as I understand, Postgres does not provide too many guarantees when it comes to interesting disk failures?

I don't mean to advocate for cryptocurrency in any way.

I'm genuinely curious, because I would have thought that a more modern distributed database that guarantees strict serializability in the face of network partitions would be the more obvious choice for tracking account balances safely?

> multi-region durability

Why is this important? Why can't they get by with a traditional main server plus some backups with a sync, plus failover when the main server goes down? It seems to me that all of your questions are issues that have been often discussed over the last 20 years, and they all have well known answers, except perhaps "multi-region durability". Leave that out, and you've well known answers for your other questions. And it seems to me, what they are saying is that they found it easier to go with a boring and traditional setup, that is, a boring and traditional setup is easier than any other setup.

"Why is this important?"

Because without multi-region or multi-AZ durability, Wave could lose balances. For example, in the event of a DC fire.

"Why can't they get by with a traditional main server plus some backups with a sync"

Because without distributed strict serializability, Wave would also be risking the loss of data back to the last backup.

However, I'm assuming here that Wave are running consensus around Postgres. That they're not taking any of these chances.

But if they're doing it right, then Postgres just doesn't seem like the most boring way to get distributed failover safely, especially when the data is as valuable as account balances, where so much compliance is at stake. Why not simply use a distributed database to begin with?

Not losing transactions with a database is not exactly terra incognita.

> Because without distributed strict serializability

Depends what you mean by "distributed". A simple 2-phase commit would do it. Or hell, just a write-ahead log on the application layer.

> Postgres just doesn't seem like the best way to get distributed failover safely

That may be so. But are you arguing in favour of blockchain? When was the last time a real bank forgot your bank balance? They never used blockchains for this.

Pretty regularly [1], depending on the bank. Every year, there will be several hour-long outages or out right data loss incidents (not all get publicized).

https://www.wcjb.com/2020/08/05/bank-of-america-glitch-accou...

Would you also class going to pay for something and not being able to because the payment network is down as "forgetting your bank balance"? Because those happen even more frequently; the cause being the POS terminal not being able to check your balance, so naievely defaulting it to $0 and giving an error about the network not being available.

Having an outage is not the same as forgetting your bank balance, even if you see a scary $0 when you log in during the outage.

But I agree it's definitely a thing that can prevent you accomplishing the thing you want to do with your money, during the outage.

But if we're talking "outage" then bitcoin fees going over $25 is probably also fair to classify as an outage as a currency. And that happens too.

And what about the first kind of failure I mentioned, that we see pretty regularly in the news? That the banks are usually pretty good at reconstructing your balance from secondary sources, does not cover for the fact that they can still lose your balance in the first place.

>Having an outage is not the same as forgetting your bank balance

I wouldn't class a high fee as an outage. If your transaction was important enough that it warranted being settled immediately and at a higher priority than every other transaction during peak rush, then a higher fee might be a pretty good trade-off. Keep in mind that the banks do not offer this feature, so in their case your transaction would simply not go through giving you no choice in the matter.

For a $10 purchase a $62 fee is in my opinion certainly an outage.

But also there have been hours-long transaction times.

I wouldn't say that banks "lose your balance" at all. It's a failure in availability.

If the bank fails or does the extremely common cryptocurrency thing of "rugpull" then you still have your money because FDIC.

You say "immediately", but i regularly do inter-bank transfers that go through faster than any Bitcoin transaction can go through, no matter what transaction fee offered. And that's not even talking about CC payments, which are done in milliseconds.

Those transactions that appear to go through "immediately", are not settled for many days or weeks. That third parties are willing to take on the settlement risk is beside the point as the same could be offered by third parties working on top of the Bitcoin blockchain.

> Those transactions that appear to go through "immediately", are not settled for many days or weeks.

The thing about that is that end users don't care, not even a little bit. It's about as interesting as whether the banks runs Linux, BSD, or OS/400.

The service provided is immediate. I don't care if the two banks need to ship physical checks around on the backend.

> the same could be offered by third parties working on top of the Bitcoin blockchain.

But that defeats the whole point of the bitcoin blockchain.

It does not defeat the point of traditional banking.

High fees In 2017 during an explicit attack on the network with spam, such that the spammer doesn’t have much BTC anymore. I haven’t paid over 1sat/vB for years. It’s the cheapest it can be onchain. Negligible. Ethereum on the other hand is broken with it’s gas calculation model with the unit of account being worth so much. Don’t touch ethereum these days, it’s bad.

2017 is not the only time this has happened. It was $62 in Apr 2021, for example.

CC payments and bank transfers are also way faster than bitcoin, even when bitcoin isn't having an outage. In the US bank transfers are slow, for some reason, but in Europe they're not.

Bitcoin has not had an outage since 2013. Actually, you will find settlement time is actually much longer than instant SEPA, and other methods have 1 and 2 and 3 day transfer times too. The instant thing is an illusion granted by centralization.

But of course your are talking about banks because you are missing the point of the exercise.

> not had an outage

You just said 2017. I said 2021. And there's also at least 2018.

> You will find that

I will not find that, because i told you what my experience is.

> Illusion

Illusion or not, it's real. /s

> Missing the point of the exercise

The point being lying and contradicting ir sequential sentences to pump a pyramid scheme.

Cryptocurrency on is not fit for any of the stated purposes, and HN will ban us both if we continue, so i bid you good day.

"But are you arguing in favour of blockchain?"

No, please see my original comment, where I made this clear: "I don't mean to advocate for cryptocurrency in any way."

By "modern distributed database", I don't mean blockchain. I just mean "modern distributed database", e.g. things like FoundationDB, Spanner, Aurora or CockroachDB.

"A simple 2-phase commit would do it. Or hell, just a write-ahead log on the application layer."

Of course, and that leads into my second question, also in my original comment:

How are storage faults in the middle of the committed WAL handled? Or is the rest of the committed WAL simply discarded, conflated with a torn write from a system crash?

These questions are important when it comes to storing balances safely. Banks can use all kinds of techniques for defense-in-depth (not to suggest they do), they're not limited to Postgres. But a distributed database is a good thing these days, no?

Gotcha.

I'm also excited about the modern distributed databases. They're pretty awesome.

But yeah, a HN comment is not the right place to design a resilient DB setup. Like I said it's not terra incognita, but it's also not just "spin up a postgres".

"Gotcha. I'm also excited about the modern distributed databases. They're pretty awesome."

Awesome!

"HN comment is not the right place to design a resilient DB setup."

Why not?

In my experience, HN is always the place to talk about building stuff, running stuff, and how to do that better. :)

Well, one reason is "you're posting often, slow down". I tried to reply yesterday, but was blocked.

The gist of my reply yesterday was "depends on requirements, and even they will be huge. And if we just design from made up requirements that's a lot of work to produce a design that nobody will use".

> Because without multi-region or multi-AZ durability, Wave could lose balances. For example, in the event of a DC fire.

I think you are making a bunch of assumptions that you haven't written down, so it is hard for me to guess what your assumptions are. In the boring, normal, standard setup that people have been using Postgres for in the last 20 years, data does not get lost so long as you have a main server and a sufficient number of backups running in sync, possibly at some distance. A question like "How not to lose data when your main server is destroyed by a fire" is an old question that database engineers have been working on for 40 years, and for which there are many good answers.

But I think you're imagining some kind of automatic link between the distributed nature of the blockchain leading to a distributed Postgres. But you would need to write out what your assumptions actually are before I could respond to them.

"But I think you're imagining some kind of automatic link between the distributed nature of the blockchain leading to a distributed Postgres. But you would need to write out what your assumptions actually are before I could respond to them."

I think the assumption here is that I'm advocating blockchain, which I'm definitely not. Please see my original comment.

You seem to be vastly overthinking this. People have been building applications (including banking applications) using boring old relational databases for decades without losing data. Single master, replicated to another region. If relational databases were as faulty as you describe, every web app you use would be losing data all the time. Not everything has to be an overthought system design interview question.

Single DB corruption and disk failures are more common than you think.

"Single master, replicated to another region."

Why do you recommend against using a modern distributed database?

Is it that hard to spin up something like CockroachDB? In fact, it wouldn't surprise me that Wave are running CockroachDB as their Postgres foundation under the hood.

> Why do you recommend against using a modern distributed database?

Call me maybe. This is a newer technology and the growing pains have been dramatic. I assume you've followed Jespen?

https://aphyr.com/tags/jepsen

I use MongoDB all the time, for a lot of projects. But I'm also keenly aware of how many years they have struggled to pass the Jespen tests. I can also recall several times engineers from MongoDB have said "We have finally resolved all of the issues raised by Jespen" and then another bug was discovered.

Again, I use MongoDB all the time. But I'm aware it is simpler and safer to use Postgres, for many, many use cases.

"I assume you've followed Jespen?"

Yes!

You might also want to check out autonomous deterministic testing, which is like Jepsen, but where you can explore many more state spaces by speeding up time, and where you can replay any distributed bugs instantly, just by replaying a seed.

This is a brilliant talk on these techniques:

FoundationDB — How I Learned to Stop Worrying and Trust the Database — https://www.youtube.com/watch?v=OJb8A6h9jQQ

I also work on a distributed database called TigerBeetle [1] where we use these new techniques. You can run our simulation tests yourself even. It's as simple as cloning the repo and running "scripts/vopr.sh", and let me know if you want a tour!

These testing techniques work so well that we've been running a bug bounty challenge with awards up to $8192 for anyone who can find a way to break it or crash it. It comes with the simulation testing all included so it's pretty easy to get started. [2]

[1] https://github.com/coilhq/tigerbeetle

[2] https://github.com/coilhq/viewstamped-replication-made-famou...

I'm not sure "struggled to pass" is the right characterisation. The Jepsen tests are part of the MongoDB test suite. They have found some bugs in the past and we have fixed them and incorporated into our test suite. Am I about to say Jepsen will never find another bug in MongoDB? No. But if they do we will fix those as well.

And I understand that MongoDB are also testing with Antithesis [1] and getting lots of assurance from that, right?

That gives me confidence.

[1] https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-A...

I followed MongoDB and Jespen in detail for years and I think "struggle to pass" is a correct characterization, which I believe even Kyle Kingsbury used it at one point. There was a long stretch where MongoDB treated Kingsbury as an enemy, and simply ignored him, or said he was wrong, then there were some years where they were saying "we dealt with all of the issues that Kingsbury raised" and then Kingsbury would write another blog post saying "Well, I still find these issues." Finally, MongoDB changed gears and decided to work with Kingsbury in a productive way, and then Jespen began to become part of the MongoDB test suite.

Reply on news.ycombinator.com