philbe77

TypeScript: https://www.npmjs.com/package/@quack-protocol/sdk

Two new Quack client drivers:

ADBC: https://github.com/gizmodata/adbc-driver-quack JDBC: https://github.com/gizmodata/quack-jdbc

Now it is easy to use Power BI with a DuckDB server you can self-host: GizmoSQL. Powered by Arrow Flight SQL - you can run terabyte-scale analytics for dirt-cheap...

Oh, and GizmoData Cloud (SaaS option) is coming soon - to make it easier than ever to provision GizmoSQL instances...

GizmoSQL is definitely a good option. I work at GizmoData and maintain GizmoSQL. It is an Arrow Flight SQL server with DuckDB as a back-end SQL execution engine. It can support independent thread-safe concurrent sessions, has robust security, logging, token-based authentication, and more.

It also has a growing list of adapters - including: ODBC, JDBC, ADBC, dbt, SQLAlchemy, Metabase, Apache Superset and more.

We also just introduced a PySpark drop-in adapter - letting you run your Python Spark Dataframe workloads with GizmoSQL - for dramatic savings compared to Databricks for sub-5TB workloads.

Check it out at: https://gizmodata.com/gizmosql

Repo: https://github.com/gizmodata/gizmosql

good point :) - we can re-aggregate HyperLogLog (HLL) sketches to get a pretty accurate NDV (Count Distinct) - see Query.farm's DataSketches DuckDB extension here: https://github.com/Query-farm/datasketches

We also have Bitmap aggregation capabilities for exact count distinct - something I worked with Oracle, Snowflake, Databricks, and DuckDB labs on implementing. It isn't as fast as HLL - but it is 100% accurate...

hi sdairs, we did store the data on the worker nodes for the challenge, but not in memory. We wrote the data to the local NVMe SSD storage on the node. Linux may cache the filesystem data, but we didn't load the data directly into memory. We like to preserve the memory for aggregations, joins, etc. as much as possible...

It is true you would need to run the instance(s) 24/7 to get the performance all day, the startup time over a couple minutes is not ideal. We have a lot of work to do on the engine, but it has been a fun learning experience...

This is something we are trying to take a novel approach to as well. We have a video demonstrating some TPC-H SF10TB queries which perform inner joins, etc. - with GizmoEdge as well: https://www.youtube.com/watch?v=hlSx0E2jGMU

I've since learned (from a DuckDB blog) - that DuckDB seems to do better when the XFS filesytem. I used ext4 for this, so I may be able to get another 10 to 15% (maybe!).

DuckDB blog: https://duckdb.org/2025/10/09/benchmark-results-14-lts

Hi mosselman, GizmoEdge is not open-source. DeepSeek has "smallpond" however, which is open-source: https://github.com/deepseek-ai/smallpond

I plan on getting GizmoEdge to production-grade quality eventually so folks can use it as a service or licensed software. There is a lot of work to do, though :)

hi djhworld. The 5s does not include the download/materialization step. That parts takes the worker about 1 to 2 minutes for this data set. I didn't know that this was going on HackerNews or would be this popular - I will try to get more solid stats on that part, and update the blog accordingly.

You can have GizmoEdge reference cloud (remote) data as well, but of course that would be slower than what I did for the challenge here...

The data is on disk - on locally mounted NVMe on each worker - in the form of a DuckDB database file (once the worker has converted it from parquet). I originally kept the data in parquet, but the duckdb format was about 10 to 15% faster - and since I was trying to squeeze every drop of performance - I went ahead and did that...

Thanks for the questions.

GizmoEdge is not production yet - this was just to demonstrate the art of the possible. I wanted to divide-and-conquer a huge dataset with a lot of power...

If I used "spot" instances - it would have been 63 x $0.732/hr for a total of: $45.99 / hr.

I'm not exactly sure yet. My goal was to not have the shards be too large so as to be un-manageable. In theory - I could just have had 63 (or 64) huge shards - and 1 worker per K8s node, but I haven't tried it.

There are so many variables to try - it is a little overwhelming...

:D that is scary!

You should do it then, and post it here. I did do it with one machine as well: https://gizmodata.com/blog/gizmosql-one-trillion-row-challen...

Challenge accepted - I'll try it on a 4XL Snowflake to get actual perf/cost

Cost-wise, 64 4xl Snowflake clusters would cost: 64 x $384/hr - for a total of: $24,576/hr (I believe)

Good point - I'll update it...

Hi nodesocket - I tried to do 4 CPUs per node, but Kubernetes takes a small (about 200m) CPU request amount for daemon processes - so if you try to request 4 (4000m) CPUs x 16 - you'll spill one pod over - fitting only 15 per node.

I was out of quota in Azure - so I had to fit in the 63 nodes... :)

At 5 seconds - the query technically cost: $0.3276

Yes - I also had GizmoSQL (a single-node DuckDB database engine) take the challenge - with very good performance (2 minutes for $0.10 in cloud compute cost): https://gizmodata.com/blog/gizmosql-one-trillion-row-challen...

A Standard E64pds v6 costs: $3.744 / hr on demand. At 63 nodes - the cost is: $235.872 / hr - still cheaper than a Snowflake 4XL cluster - costing: 128 credits / hr at $3/credit = $384 / hr.

Hi MobiusHorizons, I happened to use websockets b/c it was the technology I was familiar with. I will try to learn more about normal sockets to see if I could perhaps make them work with the app. Thanks for the suggestion...

Hi shinypenguin - the dataset and challenge are detailed here: https://github.com/coiled/1trc

The data is in a publicly accessible bucket, but the requester is responsible for any egress fees...

Here is a video demo I uploaded: https://youtube.com/watch?v=bYmFd8KBuE4&si=YbcH3ILJ7OS8Ns47

Hi, I'm Philip Moore - founder of GizmoData (https://gizmodata.com).

I recently posted on Show HN about my company's product: GizmoSQL - here: https://news.ycombinator.com/item?id=43427671

Since posting that, I wanted to make it easier for folks to "kick the tires" of GizmoSQL at https://app.gizmodata.com

There you can run interactive SQL queries against a 1TB TPC-H database file. The "lineitem" table has nearly 6 billion records...

Right now, I just have a single Azure Cobalt 100 (arm64) VM running the database (I have a very small company) - so I may scale this up (or shut it down) - depending on usage/interest.

I built the front-end GUI with help from lovable.dev (I'm typically a back-end engineer). The middle-tier is a Python-based websocket proxy server I built for Arrow Flight SQL - which makes it easy to hook up a javascript frontend (i.e. a browser) to an Arrow Flight SQL database server (which GizmoSQL is). I haven't decided if I want to open-source this piece yet, but I am thinking about it...

---

Here are some more details on GizmoSQL - what it is, and what it does:

GizmoSQL allows you to run DuckDB or SQLite as a server (remotely) - harnessing the power of computers in the cloud - which typically have more CPUs, more memory, and faster storage (NVMe) than your laptop. In fact, running GizmoSQL on a modern arm64-based VM in Azure, GCP, or AWS allows you to run at terabyte scale - with equivalent (or better) performance - for a fraction of the cost of other popular platforms such as Snowflake, BigQuery, or Databricks SQL.

GizmoSQL is self-hosted (for now) - with a possible SaaS offering in the near future. It has these features to differentiate it from "base" DuckDB:

Run DuckDB or SQLite as a server (remote connectivity) * Concurrency - allows multiple users to work simultaneously - with independent, ACID-compliant sessions * Security * Authentication * TLS for encryption of traffic to/from the database * Static executable with Arrow Flight SQL, DuckDB, SQLite, and JWT-CPP built-in. There are no dependencies to install - just a single executable file to run * Free for use in development, evaluation, and testing * Easily containerized for running in the Cloud - especially in Kubernetes * Because it is powered by DuckDB - GizmoSQL can work with the popular open-source data formats - such as Iceberg, Delta Lake, Parquet, and more.

GizmoSQL performs very well (when running DuckDB as its back-end execution engine) - check out our graph comparing popular SQL engines for TPC-H at scale-factor 1 Terabyte - on the homepage at: https://gizmodata.com/gizmosql - there you will find it also costs far less than other options.

We would love to get your feedback on the software - it is easy to get started for free in two different ways:

For a limited time - try GizmoSQL online on our dime - with the SQL Query Navigator - it just requires a quick registration and sign-in to get going - at: https://app.gizmodata.com - where we have a read-only 1TB TPC-H database mounted for you to query in real-time. It is running on an Azure Cobalt 100 VM - with local NVMe SSD's - so it should be quite zippy.

Download and self-host GizmoSQL - using our Docker image or executables for Linux and macOS for both x86-64 and arm64 architectures. See our README at: https://github.com/gizmodata/gizmosql-public for details on how to easily and quickly get started that way

Thank you for taking a look at GizmoSQL. We are excited and are glad to answer any questions you may have!

Public facing repo (README): https://github.com/gizmodata/gizmosql-public?tab=readme-ov-f... HomePage: https://gizmodata.com/gizmosql ProductHunt: https://www.producthunt.com/posts/gizmosql?embed=true&utm_so... Try GizmoSQL online: https://app.gizmodata.com GizmoSQL in action video: https://youtu.be/QSlE6FWlAaM

Hi chrisjc, these are interesting thoughts. I was mainly interested in providing a server interface for DuckDB b/c of the performance I observed when running large OLAP workloads, and especially the cost savings vs cloud SaaS offerings. I think the big benefit here is that you can run the same workload at roughly equivalent performance - but for 80% less - from my tests...

I like your ideas, though. The server is written in C++.

I hope this helps...

Thanks!

Hi emthrowaway123 - I tried comparing to ClickHouse by signing up for a free trial - and attempting to run the TPC-H 22-query benchmark with Scale-Factor 1TB against data stored in S3.

I repeatedly ran out of memory during benchmark query execution when attempting to use a 3-node cluster - with the biggest nodes my trial would allow.

I suspect that ClickHouse may not support "out-of-core" processing, b/c of the memory issues I experienced, but I'm not a ClickHouse expert by any means. I may have configured something incorrectly - but it was definitely not a turn-key experience to try to get up and running with a 1TB benchmark.

I had tried a few months ago, so I should likely try again sometime with a newer version of ClickHouse.

Thanks for the kind words!

I think the main advantages this has over other RDBMS's such as Postgres and MySQL is performance - due to the columnar / vector processing capabilities of DuckDB. I've run this in AWS with a i8g.16xlarge, and on Azure with a Standard_E64pds_v6 - and get amazing performance - due to the use of NVMe storage, lots of CPU (64) and memory (512GB) - for less than $4/hr in cloud VM cost.

This solution lets users use the large resources available on larger cloud VMs as an enterprise-grade server - for use with data notebooks, analytics dashboards, and more.

You can get performance that meets and exceeds many distributed systems on a much simpler architecture - reading from parquet datasets, assuming you've copied (cached) them on the local NVMe SSD storage. Of course - this has disadvantages - such as keeping the local copy in sync with the cloud storage (S3) - but it can be well worth it if you have a mostly read-only dataset (write-once, read-many). There is a small startup time penalty for hydrating the local storage - but I've seen throughput as high as 4GB/s for this initial copy...

I hope this helps. Thanks again for the question!

Hey mousematrix, I've tested with 3 concurrent sessions all writing to the same table - and with "BEGIN TRANSACTION" for each.

I opened a TPC-H SF1GB database with GizmoSQL, started 3 concurrent client sessions, and had each session run:

BEGIN TRANSACTION; INSERT INTO lineitem SELECT * FROM lineitem; COMMIT;

None of the 3 sessions were blocked - and all were able to insert into the table, and read the results of their inserts before committing.

It was very cool to see also that writes do NOT block reads from other sessions - the other sessions just could not see uncommitted data.

After committing in one session, the other sessions could see the data that was inserted.

I haven't stress tested the system to get limits, but I thought this little test would prove useful information for your question.

I'll work to do some benchmarking on concurrent inserts, etc.

Updates to the same data by multiple sessions behave a little differently - see details from page: https://duckdb.org/docs/stable/connect/concurrency.html

Thanks for your question!