Hacker News Reader

This looks very interesting. I've got multiple PB of log files that I'd like to be able to do real-time and batch processing on. Log files are being generated at a small number of Gbps.

Anyone else got that much data and have a good solution?

View on HN

I don't think anyone has managed to get interactive speeds (~seconds to max couple of minutes) at the PB scale.

What kind of analysis are you trying to do on that dataset? Log files can be verbose as well. Do you need all the data in the logs or can you cut down the data in each row to only what you need?

Reply on news.ycombinator.com