Hacker News Reader

Comment by sandGorgon | Hacker News Reader

This is awesome. I can't seem to under why most blockchains insist on using leveldb as their transactional data store. In fact, the bitcoin devs have admitted some of the attacks are specifically targeted towards the non ACID nature of leveldb.

It would be so much better to use postgresql as the data store. But it seems there is a natural aversion to non-embedded databases.

View on HN

You want your client to be standalone, you can't expect users to install and configure postgresql and sqlite probably isn't gonna cut it.

If you want blockchain stats, postgres is great, but for a node you just need to be able to look stuff up by its keys. Leveldb was made for that and it's good at doing it.

I don't know these attacks, but it seems you don't really need ACID that badly, most stuff is append only and once appended it never gets deleted.

I'm curious, why wouldn't sqlite cut it? Granted, I'm not a blockchain afficiando, but I have several sqlite DBs that run a couple hundred gigabytes a piece that work quite well.

I said probably because I thought about it myself while writing that comment. I also have good experience with it, but if sqlite only talks SQL then just creating string representation of binaries would already be prohibitive.

> but if sqlite only talks SQL

What does that mean? Like any RDBMS, SQLite is interfaced with in SQL, but can store and process binary data just fine.

I mean that you need to serialize binary just to make an SQL statement that includes it, and that's very expensive. E.g. postgresql's COPY supports binary format which allows you to avoid that.

The fact that SQLite is embedded means it can take full advantage of pointer passing. I don't think it serializes anything when you bind parameters to a blob.

FWIW, you can use the binary protocol for SELECTs too.

SQLite would cut it just fine. Though it would also need an extension like this one, if you'd want the signature verification and hashing to happen inside SQLite.

The difference between SQLite and Postgres would be that more than one client can connect over the network to PG (or most any other sql db out there really), while SQLite is meant to be embedded, which, if you already have LevelDB working (as is the case in Bitcoin Core), wouldn't be all that different.

I'm not sure why that's an argument. Why can't you expect?

Its like saying WordPress users should not setup a database. There is a transactional usecase for this - having a high performance, ACID datastore.

And you are mistaken: here's a comment by achow101 of Bitcoin Core - https://bitcointalk.org/index.php?topic=1394020.msg14159449#...

LevelDB being stupid is one of the major reasons that people have to reindex on Bitcoin Core crashes. There have been proposals to replace it but so far there are no plans on doing so. However people are working on using different databases in Bitcoin Core and those are being implemented and tested.

https://www.reddit.com/r/Bitcoin/comments/6z776p/breaking_bi...

What doesn't help this is LevelDB's transaction format: when you update the utxo set, you want to do this atomically (otherwise you may end up with a corrupted state). To do so, you start a DB transaction (do not confuse this with Bitcoin tx, this is just how you build a change set before pushing it to disk), update the change set then commit the tx. With LevelDB, as long as the transaction is live, the entire changeset will remain in RAM (usually a DB engine writes the changes on disk as you provide them, then updates internal pointers at commit time to apply the change), basically double dipping on the RAM hogging and I/O inefficiency

Why would it be better to use Postgres?

It implements the SQL standard very well It includes support for "advanced" SQL stuff like window functions or common table expressions It is very innovative in terms of how plpgsql interacts with SQL It supports lots of advanced data types, such as (multi-dimensional) arrays, user-defined types, etc.

These are pretty good reasons :)

Which of those are relevant to implementing a blockchain node?

Two reasons : ACID and single node scalability. Postgresql is production tested with terabytes of Ram and disk usage on a single node.

It has the jsonb column that can easily replicate the exact data model that you would use in leveldb.

However there's another significant reason - devops if you are looking to do anything serious with blockchain. There is really no good way to build a meaningful production application on the blockchain without serious devops because of leveldb. What is funny is that everyone who wants to do something on the blockchain first pulls the data into postgresql and then runs something on it.

Why have this problem in the first place? It just adds a few minutes to setup if you use postgresql instead of leveldb and get so so much more. And suddenly you can leverage all cloud hosted DB (rds, etc) and not have to focus on devops

What is this "devops" you're talking about?

Reply on news.ycombinator.com