Hacker News Reader

Did you try out DuckDB?

I considered it, but when I started the project I was using a Supabase DB for the backend since their free-tier is quite nice, and after the switch to and from BigQuery, PostgreSQL was the easier migration. I also thought Postgres might be more suited to the backfill/ingestion job due to the frequent row-level reads and writes. That said, I know DuckDB can use Postgres in the backend, and I might consider that if the current model starts to struggle.

I have also seen some benchmarks that suggest the gap between DuckDB and Postgres isn't always so substantial: https://jsonbench.com/#eyJzeXN0ZW0iOnsiQ2xpY2tIb3VzZSI6dHJ1Z...

What makes performance on JSONBench relevant to this project? Are you storing and querying large JSON object blobs in the database?

That's a fair question, I shouldn't have posted that without context. I was initially considering storing JSON to speed up the ingestion since the results are in JSON, which is how I found JSONBench. But, of course, parsing the JSON isn't the bottleneck for me - rate limits and response time is - so I didn't end up going that route.

What I mention first in my message was a much bigger driver - convenience. If the analytics become much more complex I might revisit DuckDB or another OLAP solution.

Reply on news.ycombinator.com