Design and implementation of DuckDB internals | Hacker News Reader

Design and implementation of DuckDB internals

duckdb.org

204 pointsmpweiher17 comments

If you are a data scientist or do anything with data... duckdb is like a swiss army knife. So many great ways it can help your workflow. The original video from CMU in 2020 [1] is a classic. Minutes 3-8 present a good argument for adding duckdb to your data cleaning/processing workflow.

And if you want to add a semantic layer on top of data, Malloy [2] is my favorite so far (it has duckdb built in):

[1]: https://www.youtube.com/watch?v=PFUZlNQIndo [2]: https://docs.malloydata.dev/documentation/

Thank you for the recommendation on that video! I've already adopted to using DuckDB for my ad-hoc analytics work but I didn't know the background

Analytics with type-safe raw SQL (including DuckDb’s awesome extensions) is pure gold:

https://github.com/manifold-systems/manifold/blob/master/doc...

Over the years I've seen anecdotes here on HN that DuckDB crashes often for several people. Is this still an issue for anyone?

We use it heavily at my workplace. It doesn't crash at all if you use it as OLAP. But if you use it incorrectly, it will crash.

It's pretty solid.

I use DuckDB daily.

In short — It doesn’t crash often at all.

What you may be remembering were reports of exceptional cases where it didn’t handle out of memory errors well. I was one of the people affected. I was running complex analytic queries on 400 GB parquets and I only had 128GB memory. It used jemalloc which didn’t gracefully degrade. They fixed a lot of the OOM issues so it’s more robust now. I haven’t had a crash for a long time.

On normal sized datasets it never crashes.

Never seen this and have several products that use it...

The actual slides are linked from the intro-text:

https://github.com/DBatUTuebingen/DiDi

That was a good start for understanding DuckDB internals!!!

Unfortunately it does not seem that there are lecture videos.

Am I missing something or is the content empty?

https://github.com/DBatUTuebingen/DiDi

Thank you, I didn't realize all of the course counted as "slides and auxiliary material" haha

edit: Really great stuff in here. Every day at work I think about how much I love DuckDB

What do you use it for? What's the best part for you?

Computing page rank on par with NetworkX: https://github.com/idesis-gmbh/WikiExperiments Educational local DW from Github Archive events: https://github.com/idesis-gmbh/GitHubExperiments

It is quite fast for OLAP applications. It works on low cost hardware.

Sorry I missed this reply.

I'm currently using it to provide a SQL interface for doing data analytics for customers that don't have data large enough to justify a large platform like Snowflake or Databricks. Its syntax is close enough to the other two that, especially since the logic is already normalized through abstracted query definitions, it's a drop in replacement.

Given that it's so lightweight, I can use it run searches in an AWS Lambda function, which is crazy useful.

thank you! Learned why DuckDB is named this way

Reply on news.ycombinator.com