Aren't log files processed a line at a time? Last time I had to deal with some structured log, I streamed lines concurrently into json parser and it went pretty fast.