DuckDB 1.0.0
duckdb.org
duckdb.org
This seems like an excellent structure for long-term protection of the open source project. What other projects have taken this approach?
DuckDB Labs: The core contributors. Instead of developing features that will be behind a paywall, they provide support and consulting. Awesome.
DuckDB Foundation: A non-profit that ensures DuckDB remains MIT licensed. Perfect.
We actually just published a post on how to replace your warehouse with DuckDB. It's certainly not a good move for every company using something like Snowflake, but it was the right move for us.
I remember this was an issue at an ESOP I worked for. They had to pressure former employees and buy back shares from them. If you have too many shareholders in the US then your corporate status changes
Granted this is a nonprofit so maybe the rules are different, but the fundamental problem remains
I love tools that make life simpler. I've been toying with the idea of storing 1TB of data in S3 and querying it using duckdb on an EC2. That's really old/boring infrastructure, but is hugely appealing to me, since it's so much simpler than than what I currently use.
Would love to hear of others' experiences with duckdb.
1 TB is .. not a lot of data .. I rent a server with 15 TB for ~50$ per month and buying a new 2TB disk is less than 100$.
Disclaimer: I work at DuckDB Labs.
[1] https://duckdb.org/docs/data/parquet/overview#partial-readin...
Since the data in S3 receives updates every few hours, querying it directly ensures queries are on up-to-date data, whereas creating a copy of the dataset on the EC2 would necessitate periodically checking for updates and moving the deltas across to the EC2 (not insurmountable; but complexity worth avoiding if possible).
I thought it was a great way to give analysts direct access to data to do adhoc queries while they were getting familiar with the data.
(Co-founder)
In combination with sqlalchemy, it is trivial to lift an app to other OLAP systems.
[1]: https://duckdb.org/docs/api/python/overview.html#dataframes
Due to policies at work it's unlikely we would use this in production, but as I understand it, it's still pretty useful for exploring and poking around local data. Is that right? Does anyone have examples of problems they've used it on to digest local files or logs or something?
Your blog does a great job contrasting the two use cases. I don't think too much has changed on your main use case, however here are a few ideas to test out!
DuckDB can read SQLite files now! So if you like DuckDB syntax or query optimization, but want to use the SQLite format / indexes, that may work well.
Since DuckDB is columnar (and compressed), it frequently needs to read a big chunk of rows (~100K) just to get 1 row out and decompressed. Mind trying to store your data uncompressed? Might help in your case! (PRAGMA force_compression='uncompressed')
By the way, I like your blog's style! Even the html of an article is clean and readable.
(From your blog post). When values span orders of magnitude, that’s when log plots are useful.
Curious if anyone else is using DuckDB for something similar? Does anyone have an example?
Does anybody know how the DuckDB foundation works? The sponsors are MotherDuck, Voltron, and Posit, which are heavily venture-funded. Do DuckDB Lab employees work on sponsored projects for the foundation?
I am also curious if anyone can shed light on what kind of contract work DuckDB does to align its work with the open source project. This has always seemed like the holy grail, but it is difficult to do in practice.
It also works seamlessly with Pandas in Jupyter notebooks. You can query dataframes seamlessly and store query results back into dataframes for plotting etc.