Mining metrics from unstructured logs

What interests me about this method is that they are finding the structure by rejecting it.

That is, somebody could say there is structure latent in the log because the lines were all generated by a template so one could write a bunch of regexes that parse them and extract structured data.

It seems they are using heuristics to remove the variable parts of messages leaving behind mostly the fixed text of the message which they then hash which lets them count that a certain kind of message was generated 77 times in the last hour.

It would be nice to turn this into a system that automatically finds the variable slots and writes the rules to turn this into more structured data.