If anyone here is using DuckDB in production i'd love to hear what your stack looks like over the entire lifecycle of extract->transform->load.
Essentially it's: 1. Data sources from places such as s3, sftp, rds 2. Use duckdb to load most of these with only extensions (I dont believe there's one for sftp, so we just have some python code to pull the files out.) 3. transform the data however we'd like with duckdb. 4. convert the duckdb table to pyarrow 5. Save to s3 with delta-rs
FWIW, we also have this all execute externally from our orchestration on an EC2 instance. This allows us to scale vertically.
Last time I checked duckdb didn't have the concept of a metastore so do you have an internal convention for table locations and folder structure ?.
What do you use for reports/visualizations? notebooks ?.
Reports and viz varies a lot, the finance department uses tableau where as for more 'data sciencey' stuff we normally just use notebooks.
My only work with it to date has been to load-analyze-usuallydelete to refresh, so I do not require any db mutability. Outside of mutability, not sure if there are any obvious wins with the format.
My stack is now JSON files containing the search data. At present I'm loading these into DuckDB and pre-aggregating due to a bug I found in DuckDB, not because it was needed for performance.