Hacker News Reader

Comment by tetris11 | Hacker News Reader

> DuckDB is designed from day one to use all the CPU cores in your machine

Hi. No thank you? I'm tired of greedy software that I have to literally pin down by shackling it to a cgroup via systemd or docker.

1 core is fine. We have a group of 20, and limited resources, and there are no fires.

View on HN

You can limit the number of threads used by DuckDB via `SET threads = x`, see https://duckdb.org/docs/configuration/pragmas#threads

(Disclaimer: I work at DuckDB Labs)

Why can't the scheduler just do its job and let applications enjoy the highest performance they can achieve? I assume DuckDB is making good use of those cores, not just spinning wildly ;).

True, the scheduler usually does do fine job CPU-wise. In terms of RAM though, if an app sees 120GB it will expand to 120GB before considering paging.

I'm just not a fan of software that immediately tries to use all possible resources for 2% gain in a performance benchmark over peers.

That’s not how most apps work. They tend to allocate memory for specific purposes and free it when done. Only more heavy apps will retain large buffer pools, like databases but they are usually tunable. And apps themselves don’t page to disk, usually. The OS does that. Before we had MMUs apps like photoshop used to manage the page file manually. But again an app doing that is doing that for a specific purpose, not just to be greedy.

Is there some specific software you are referring to?

No, but a general rant at analysis software in general which seeks to maximise all resources.

I agree that database software likely does not behave so aggressively, but the terms I saw in the duckdb post triggered me to believe they were trending in that direction.

I think having an option to utilize multiple cores is better than not having one.

Would be nice to have an option to limit the number of cores used by DuckDB though, if it doesn't have one already. (`duckdb -j4`?)

Yeah, this is kind of a nuisance on high-core count CPUs, especially SMP.

Reply on news.ycombinator.com