Adding concurrent read/write to DuckDB with Arrow Flight

You can sort of see what benefits you might get from a post like this, though: https://willayd.com/leveraging-the-adbc-driver-in-analytics-...

While we're not using Arrow on the wire here, the ADBC driver uses Postgres's binary format (which is still row oriented) + COPY and can get significant speedups compared to other Postgres drivers.

The other thing might be to consider whether you can just dump to Parquet files or something like that and bypass the database entirely (maybe using Iceberg as well).

The concurrency rules of DuckDB is here [1]. Reads/writes need to happen in the same process, but multiple threads can do so.

This is putting a server in front of a DuckDB instance, so all read/writes are funneled there in that one process. DuckDB takes care of the concurrency within the process via MVCC.

You could do the same thing with an HTTP server or other system, but this shows it with the Flight RPC framework.

NOTE: I had an incorrect comment for about 2 minutes that I deleted and restructured here. Sorry if you saw that noise.

[1] https://duckdb.org/docs/connect/concurrency

https://github.com/cosmos/iavl :

> Merkleized IAVL+ Tree implementation in Go

> The purpose of this data structure is to provide persistent storage for key-value pairs (say to store account balances) such that a deterministic merkle root hash can be computed. The tree is balanced using a variant of the AVL algorithm so all operations are O(log(n)).

Integer Vector clock or Merkle hashes?

Why shouldn't you store account balances in git, for example?

Or, why shouldn't you append to Parquet or Feather and LZ4 for strongly consistent transactional data?

Centralized databases can have Merkle hashes, too;

"How Postgres stores data on disk" https://news.ycombinator.com/item?id=41163785 :

> Those systems index Parquet. Can they also index Feather IPC, which an application might already have to journal and/or log, and checkpoint?

DLT applications for strong transactional consistency sign and synchronize block messages and transaction messages.

Public blockchains have average transaction times and costs.

Private blockchains also have TPS Transactions Per Second metrics, and unknown degrees of off-site redundancy for consistent storage with or without indexes.

Blockchain#Openness: https://en.wikipedia.org/wiki/Blockchain#Openness :

> An issue in this ongoing debate is whether a private system with verifiers tasked and authorized (permissioned) by a central authority should be considered a blockchain. [46][47][48][49][50] Proponents of permissioned or private chains argue that the term "blockchain" may be applied to any data structure that batches data into time-stamped blocks. These blockchains serve as a distributed version of multiversion concurrency control (MVCC) in databases. [51] Just as MVCC prevents two transactions from concurrently modifying a single object in a database, blockchains prevent two transactions from spending the same single output in a blockchain. [52]

> Opponents say that permissioned systems resemble traditional corporate databases, not supporting decentralized data verification, and that such systems are not hardened against operator tampering and revision. [46][48] Nikolai Hampton of Computerworld said that "many in-house blockchain solutions will be nothing more than cumbersome databases," and "without a clear security model, proprietary blockchains should be eyed with suspicion." [10][53]

Merkle Town: https://news.ycombinator.com/item?id=38829274 :

> How CT works > "How CT fits into the wider Web PKI ecosystem": https://certificate.transparency.dev/howctworks/

From "PostgreSQL Support for Certificate Transparency Logs Now Available" https://news.ycombinator.com/item?id=42628223 :

> Are there Merkle hashes between the rows in the PostgreSQL CT store like there are in the Trillian CT store?

> Sigstore Rekor also has centralized Merkle hashes.