Hacker News Reader

Comment by faizshah | Hacker News Reader

I was curious how long it would take with Polars (for scale), apparently 33s: https://github.com/Butch78/1BillionRowChallenge/tree/main

I’m kind of interested in the opposite problem, what is the simplest solution using a well known library/db that approaches the fastest hand optimized solution to this problem?

View on HN

That’s the question worth asking imo. I was wondering how fast is the idiomatic Java solution

71 seconds

https://questdb.io/blog/billion-row-challenge-step-by-step/

Is that the same hardware? Otherwise it doesn’t say much.

Java is often slightly faster than Go, has similar (perhaps, older, better optimized Map) constructs, perhaps better GC (older, more optimized), though I don't think the GC is a challenge, has slower startup times - so I'd say roughly the same as the idiomatic Go version?

Java’s GC is just incomparably better, literally every research on the topic is written for Java. Though that’s true that without value types, Java does rely slightly more on it.

It could be matter of fact comment if incomparably better and slight reliance are brought to the middle.

Java can be faster than Go, but it comes at the same cost as most "faster" things in software: Java uses significantly more memory.

Java is also more mature, which means you are entering a massive package-bloat setup that has evolved over the years to work for everyones wild and varied needs. By the time you have your database, cache, http/other handlers, tests, fixtures, metrics, logging, tracing, etc... setup you're looking at a scary pile of dependencies spanning thousands of classes that would make even NPM jealous.

On my machine the simple/idiomatic "baseline" Java solution (https://github.com/gunnarmorling/1brc/blob/main/src/main/jav...) takes 2m0s, compared to 1m45s for my simple Go version.

Where’s the source data I’d like to attempt ingesting this and processing it with DuckDb.

In the original 1BRC, it's a python script that generates the data.

Instructions on how to create it can be found here:

https://github.com/gunnarmorling/1brc?tab=readme-ov-file#run...

The Python version:

https://github.com/gunnarmorling/1brc/blob/main/src/main/pyt...

Here’s a thread on results with duckdb, I don’t mean to discourage you taking a shot at all though: https://github.com/gunnarmorling/1brc/discussions/39

Sounds very reasonable. In the blog post about 20s were shaved off by assuming we don't need complicated string parsing. An of the shelf library can't make that assumption so they will always have to pay the extra cost.

True in general but some (especially libs aimed at larger datasets processed in batch) are taking advantage of benchmarks like this to do things like: Try the fast way, if it works great Try the slow way, if above fails This makes the slow path 2x slower at worst (and you can advise to always use the slow way with optional params) but the fast path can be 10x faster

I’m surprised at the poor performance of python here. For reference there are several very brief R examples which are just 2-3 seconds. Eg http://blog.schochastics.net/posts/2024-01-08_one-billion-ro...

Are you talking about the 2nd table in the Benchmark section?

It seems they are not running against the full dataset:

> Moving on to the 100 million file to see if size makes a difference.

  ggplot2::autoplot(reorderMicrobenchmarkResults(bench1e8))

One would also have to run both approaches on the same hardware for a meaningful comparison?

Reply on news.ycombinator.com