Other than motherduck, is anyone aware of any good models for running multi-user cloud-based duckdb?
ie. Running it like a normal database, and getting to take advantage of all of its goodies
ie. Running it like a normal database, and getting to take advantage of all of its goodies
Where you store the .duckdb file will make a big difference in performance (e.g. S3 vs. Elastic File System).
But I'd take a good look at ducklake as a better multiplayer option. If you store `.parquet` files in blob storage, it will be slower than `.duckdb` on EFS, but if you have largish data, EFS gets expensive.
We[2] use DuckLake in our product and we've found a few ways to mitigate the performance hit. For example, we write all data into ducklake in blog storage, then create analytics tables and store them on faster storage (e.g. GCP Filestore). You can have multiple storage methods in the same DuckLake catalog, so this works nicely.
0 - https://www.definite.app/blog/duck-takes-flight
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountp...
the others, I haven't tried handling it in.
It also has a growing list of adapters - including: ODBC, JDBC, ADBC, dbt, SQLAlchemy, Metabase, Apache Superset and more.
We also just introduced a PySpark drop-in adapter - letting you run your Python Spark Dataframe workloads with GizmoSQL - for dramatic savings compared to Databricks for sub-5TB workloads.
Check it out at: https://gizmodata.com/gizmosql