Ask HN: How to read a terabyte flat file
For some reason - an application spews some results into a single file. What do you think would be the best way of reading (multiple) from the file?
The parsing should be simple. Lex and/or Yacc should handle very large files. If not, you can always write something by hand, once again sticking to the store-nothing principle.
I would think about it as a fold operation (http://en.wikipedia.org/wiki/Fold_%28higher-order_function%2...) over the lines in the file, where the parser state and any aggregate calculated values are stored in the accumulator, and each new line is considered in conjunction with the current state and the previous calculated values.