While the interviewer probably wanted a typical "programming answer", I wonder how large a dataset you could do by simply using the command line. That's programming, too.

Just use sort and uniq.

Or if it's slow, sort in parallel, then merge into one stream, and extract the frequencies.