Uno: a uniq like CLI tool for log data

> Looking at a sample of the logs it’s obvious that there is both repetition and structure in this data – similarly structured messages repeat with different parameters, dates, numbers, file names, etc. This seems like a great candidate for some automation, let’s see what we can do!

While "uno" is, perhaps, "cooler", logcheck [0] has been around for ages and does just this -- go through your log files, weed out all the stuff that you don't care about, and mails you anything else (which, by definition, is all the stuff you do care about).

Logcheck doesn't use "fuzziness" or uniqueness or any intelligence or anything like that -- just regexes -- but I end up with one single mail per day that only takes me a minute or two to sift through.

Additionally, logcheck is open source.

[0]: http://logcheck.org

Protip: "sed -E 's/[0-9]+/N/g'" is, by itself, what I'd call 'unreasonably effective' at converting log lines into a canonicalized form, such that each line of printf or printf-equivalent source code corresponds to only one or a few distinct output lines. Then you can pipe it into "sort | uniq" (possibly "uniq -c" if you want counts, possibly for logspam-quantification purposes), and otherwise conceivably into a "profile" of a logfile, and then you can diff between a "normal" profile and something that may be less normal.

Here's something I'd very much like: syslog(3) and anything like it that takes printf-style format strings and arguments to log the format string and the arguments in a structured way.

E.g.,

  syslog(LOG_WARN, "this thing %s didn't go well", thing);

should log something like this, using JSON just for this example to show the actual structure:

  [<timestamp>,"<hostname>",<pid>,"<ident>","<facility>",<priority>,"this thing %s didn't go well", "<thing>"]

This would make it much easier to recognize structure in log messages and extract details (syslog(3) format string arguments).

Granted, this would have to have happened 20 years ago for it to be widely available now, but still, gotta start somewhere.

That looked very interesting. But I had so search through the video and zoom in to learn how exactly to separate events that stand out from the baseline (instead of, you know, just reading that command from the webpage).

So, i did all that and downloaded the tool and run that command and found out that the feature is "experimental" and not included in the binaries. It's a bit strange: the tool got my attention, and i was ready to experiment, but i couldn't.

If the reason is that it's a paid or saas only feature, that's totally fine but i think authors should make it a bit more obvious from the web page. Otherwise, i'd suggest to put "experimental" binaries there, too.

Does it understand stacktraces, or do I have to preprocess the mutliline stacktraces into single lines?

Appears to be just

l = readline() print l while True: n = readline() if editdistance(n, l) > 30%: print n l = n

The simple tool doesn't appear to handle interleaving? Normally uniq handles that by sorting first, but this would need an edit distance aware sort first. I suppose that wouldn't be that hard to build either.

Perhaps it could be just handled with a short window in the tool.

This tool shows a lot of promise to me and I've already installed it on our stage box. Thanks!

Some of our log lines contain large json blobs that uno only ignores when I set the difference ratio >50%. Maybe you could add some sort of json detection and parsing identify and ignore similar JSON? Seems tricky but just an idea. Also, the go routine seems to panic on the linux and mac binaries when I run "uno --help".

Tools are needed that turn log analysis into more science than art. We currently use the ELK stack to analyze our logs and it works pretty well but still feels like spelunking.

Do any of you HNers have any logging tools you'd recommend?

Is there a Homebrew package for this tool? Or another similar tool?

Good idea, but it's non-Free. Should be fairly simple to recreate the ideas as a Free tool.

_