Hacker News Reader

Arrow Flight is a very neat project! How do you not end up with race conditions? Or data overwriting each other? Now that you have multiple writers. I think typically this is solved with WAL files. So maybe it's something internal to the Flight server?

View on HN

sorry, should have covered this in the post (will likely add).

neomantra nailed it: https://news.ycombinator.com/item?id=42865405

Yeah I'm confused as to what its actually doing

I do like this article a lot for showing how to do this pattern of slurping data and inserting it into a DB, in the context of Arrow Flight.

The concurrency rules of DuckDB is here [1]. Reads/writes need to happen in the same process, but multiple threads can do so.

This is putting a server in front of a DuckDB instance, so all read/writes are funneled there in that one process. DuckDB takes care of the concurrency within the process via MVCC.

You could do the same thing with an HTTP server or other system, but this shows it with the Flight RPC framework.

NOTE: I had an incorrect comment for about 2 minutes that I deleted and restructured here. Sorry if you saw that noise.

[1] https://duckdb.org/docs/connect/concurrency

Thanks! That makes sense now.

Reply on news.ycombinator.com