Why DuckDB 2.0 is faster
motherduck.com
motherduck.com
Most of the DB engines out there still seem to use a "n-threads" style parallelism with exchange operations and poor async I/O management.
DuckDB is improving on this front, but in some sense is catching up to R&D (and implementation!) that is now decades old.
A "rhetorical challenge" I like to give software developers working on systems like this is the following: If I gave you a computer with 1,024 cores and matching network and storage bandwidth -- but with significant latency -- could you keep a system like this 100% utilised with one query?
The answer for almost all software is "no".
For example, SQL Server tops out at 64 hardware threads for any one query: https://learn.microsoft.com/en-us/sql/database-engine/config...
GPU codes are starting to get there, but CPU codes are way behind on this frontier of computer science.
It's not just databases! Can you (de)compress a file in parallel? Verify its hash in parallel? Upload/download from storage with CPU and I/O task parallelism? Can you overlap all of these operation so nothing is ever waiting on anything else it doesn't have to?
This matters! I ran some tests with bioinformatics codes and found that most got stuck in tar pits. Many could not scale to modern SSDs with millions of IOPS or modern networking with hundreds of gigabits of throughput, no matter how many CPU cores were thrown at them.
PS: AMD's Zen 6 era EPYC 9006 processors will have 512 cores and 1,024 threads per two-socket system, so this is not hypothetical: https://www.amd.com/en/products/processors/server/epyc/9006-...
You can hash chunks, or maybe use different kind of hashes though.
The leaves and then all nodes can be computed in parallel.
With a typical node size of say 4 KB any file that is at least 4 MB in size can utilise 1,024 cores.
Same for objects stores like S3 or Google Drive, they only give you a single hash, you can't parallelize that, you need to add your own merkle tree on top.
Also, the tree can be based on any underlying hash function, so the root hash can be literally "just" a SHA256 hash, but instead of hashing the whole file, it hashes a list of hashes.
You don't even need to store those intermediate nodes! They must be recomputed on-the-fly to verify the root hash anyway.
Last, but not least, the temporary space required is logarithmic in the size of the data, typically with a large exponent, so in practice this all fits on the stack. No heap memory allocations required, compatible with embedded devices, and can re-use cryptographic acceleration primitives like the AES-NI instruction set.
Conversely, tree-based hash constructions allow:
- Parallel generation and verification.
- Incremental repair of partially corrupted data.
- Extension of an existing hash (efficient append & prepend).
- Patricia hashes also allow efficient mutation in the middle of the data.
- Various deduplication and efficient incremental sync scenarios.
While it's true that 2.0 makes recursive CTEs much more performant, don't forget that there's a badass DuckDB extension for graph data already. Here's a 2025 article about how elegantly that can handle what is otherwise quite a challenging class of query for your typical RDBMS: https://duckdb.org/2025/10/22/duckdb-graph-queries-duckpgq
> One setting drives this,...
> The cost is now about the rows you actually touch, not rounds times table size.
Etc.
I get the brain scramblies [1] from trying to parse this writing style at work so I hate to see it elsewhere. Apologies if I'm wrong. But if I'm not then OP don't use an LLM to write for you. It's hazardous to your reader's health [2].
[1]: https://www.youtube.com/watch?v=ipUJq-odt5Q
[2]: https://discourse.haskell.org/t/how-to-keep-enjoying-program...
> Storing a lake as thousands of 1 MB Parquet files is a bad practice anyway, and 2.0 does not rescue it.
The "does not rescue it". No human would write like that.
> I'll explain what that means on a table you already know.
No I don't already know that table.
Also
> and claims 40x on graph reachability
Is really hard to parse.
The section on recursive CTEs wasn't well written and didn't explain how the optimisation was done. This article explains how the recursive CTEs were improved https://duckdb.org/2026/08/25/how-duckdb-runs-recursive-ctes...
This is what I don’t understand. Supposedly LLMs are trained on human text. Why do they come up with such unrealistic prose? Is it intentional because the companies want the tells to be obvious?
Getting it to write well is really hard because there’s no real way to verify whether it’s good prose or not. You and I can tell, but we can’t write a verifier that codifies our judgment.
Maybe they’ll find a way to improve this, but for now it’s certainly one of the harder problems to solve for LLMs.
Part of it is that I think they also have poor theory of mind, which I imagine is also a hard thing to train it to do.
In any event, other LLMs may not automatically have the problem, and don't even require such a prompt. This is a Claude problem.
LLMs are trained on an extraordinarily diverse range of human expression, yet their default behaviour collapses much of that diversity into a surprisingly narrow conversational register. This produces a pathologically median conversational style. The result is a remarkably consistent tendency towards the middle, which works shockingly well 99% of the time. But in the remaining 1%, when the median isn't a position many actual humans would occupy, the artificiality becomes glaringly obvious.
On the contrary. It is unnatural for a native speaker, which I don't think the author is. For someone that speaks English as a second language, it is not uncommon to use expressions literally translated from their first language, which may be understandable but weird for native speakers.
> DuckDB 2.0 is coming this fall and the alpha is out! I ran the interesting features on my own laptop, and against S3, to see what actually changes for people who build tables and pipelines rather than database engines.
The first sentence looks purely ESL, the second one looks LLM.
> Because yes, DuckDB 2.0 is faster. But to get the speed bump you need to understand how your data is shaped, and sometimes how to model it.
The second sentence here looks LLM as well.
> One comment on the tiny files: no meaningful change, because the time there is per-file round trips (footer, then data) that reading ahead cannot remove. Storing a lake as thousands of 1 MB Parquet files is a bad practice anyway, and 2.0 does not rescue it. Fundamentals still matter!
This reads as LLM with no sign of ESL left.
All in all, this seems to follow the recent trend where the very beginning of the article shows the most user input while the rest is mostly generated. The only confounding factor here is the user seems ESL as well but there's still LLM all over.
For the most part I do my own writing, because AI models aren't even close to reaching human quality. But for things like dynamically generated technical documentation or automated tasks (like summarizing the results of an automated anomaly investigation), where I cannot be in the loop, I find writing can be improved by asking for a specific style.
For example, if you ask for Simplified Technical English, we go from:
Storing a lake as thousands of 1 MB Parquet files is a bad practice anyway, and 2.0 does not rescue it. Fundamentals still matter!
It is bad practice to store a data lake as thousands of 1 MB Parquet files. Version 2.0 does not correct this. The basic rules still apply.
Claude's prose causes fatigue, I think, in part because the grammatical structure is at odds with how human cognition expects to be served information in their native language. But I also suspect that's a training thing. We've not grown up reading or writing like Claude.
I mean, even the article admits that "40x on graph reachability" is difficult to parse.
I'm starting to read it as a sign of low LLM effort not just low human effort. It seems most common when one few-sentence prompt leads it to generate 4+ paragraphs (and the longer the output, the worse the odds). Prompting to dig into each resulting paragraph one by one, to make them readable, makes it do higher-effort deep dives.
There was an article on HN not long ago explaining how to get better illustrations from image generators with better prompts, and it showed that the reason why generated images all looked the same nowadays, was purely caused by poor/basic prompting.
It may be the same with text: there could be ways to get LLMs to write better prose with more specific prompting. LLMs should be able to imitate any style.
At least it emphasises the importance of truly understanding what Claude is saying. Because if you don't understand it, there's a good chance it's wrong. Do not ever think you're stupid for not understanding something Claude says.
A human has painful memories of past mistakes and knows which of their own thinking process needs to be double checked and which can be accepted on faith. An llm has no such memory, it can either verify every single word or none of them. And verification itself needs to be verified. So they have to stop somewhere.
This is why llm usually produce text that’s too shallow or too detailed.
When you ask the agent to double check something you gave it the missing piece - intuition about which part needs to be checked.
I definitely add LLMisms to my speech to other techies as ironic jokes, but I've noticed more than once it creeps in naturally. I'm starting to embrace my typos as the few lingering signs of my humanity...
I get the brain scramblies [1] from trying to parse this writing style at work so I hate to see it elsewhere
More and more really important and useful information will arrive like this for us to consume. There's no way around. So this is a disservice for newcomers that could come and go unscathed but instead is crippled by these kinds of comments that brings nothing of substance to the table and has the potential to make them hate something they otherwise wouldn't even notice.
Also, from https://news.ycombinator.com/newsguidelines.html:
Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.
Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting.Also if you think LLMisms so bad it's spam, you also don't get a free pass. From the guidelines:
If a story is spam or off-topic, flag it. Don't feed egregious comments by replying; flag them instead. If you flag, please don't also comment that you did.This sort of writing decreases readability. People say "just have your own LLM rewrite it" like you don't lose value when you go from prompt -> slop you didn't review enough to clean this crap up in -> someone else's prompt to change the style -> finally someone reads it.
Do you know what happens a double-digit percentage of the time when I ask Claude to rewrite some shit that it gives me like this? It says things like: "I overstated this, I rechecked and actually..." or "this claim doesn't hold up, actually [this other thing is true]..."
So it's a sign that the claims in the post likely weren't vetted very hard.
So if you aren't proofreading I'm gonna be skeptical. And saying "deal with it" doesn't rescue it. Does it?
And then there's the reflexive "you must just be an ideological hater." No. I'm someone who uses the tools in a domain where quality matters enough that I have to dig into the quality of the tool output and spot the tells for when it's low output, so that I can deliver shit that works reliably and consistently.
So it's a sign that the claims in the post likely weren't vetted very hard.
Just because some guy on some forum said that doesn't make it true. That's not how you establish facts regarding health claims.
Not that i question that reading mostly ai slop for long enough makes you feel dead inside.