Topfew and Amdahl

tbray.org

34 points·zdw··5 comments

> I’m using a 3.2GB file containing 13.3 million lines of Apache logfile, half from 2007 and half from 2020.

I'm not sure if this was deliberate, but I hadn't come across this use of 'logfile' as an uncountable noun (eg. like 'water', 'money', or 'data') before.

Having occasionally dug into logs over the years, it certainly feels right, though asking "how much logfile?" strikes me as a bit Doge, but I'm sure that will wear off.

Linguistic evolution in action, folks!

"Apache logfile" might be the format, just like "800MiB of json". Just a guess.

Even if it is "Apache logfile", it is still being treated as an uncountable noun.

The measure here is 'lines', analogous to 'cups of water'. Adding a format or brand so that it's 'cups of Perrier water' or 'cups of fortified water' doesn't change anything.

Neither does stating the size, any more than '2 liters of water' does.

After you shard some of the computation out to different processors to filter before single core aggregation, then you can push down some pre aggregation to limit the communication between parallel threads, and it keeps going!

Super interesting to see parallelism for reading logs this way, like for when you have too many logs for one core and too few for Spark

That turns out to be hard because in your typical log file you have bursts of requests localized to a particular part of the file that get something into the top-N without even appearing in other segments. So essentially you can end up having to do a complete N-way merge between the segments to make sure you really find the top hits.

Reply on news.ycombinator.com