Comment by a3_nm | Hacker News Reader

Comment by a3_nm | Hacker News Reader

What if I need, say, to find the 10 IPs that make the most requests? With the Apache log format, I can write the following in about 15 seconds:

  cut -d ' ' -f1 log | sort | uniq -c | sort -nr | head

Say you need to follow accesses to a particular file? The following quick and dirty one-liner probably works well enough:

  tail -f log | grep --line-buffered file.pdf

How do you do that with json?

Granted, as soon as your logs stop being a sequence of records (lines) with a fixed sequence of neatly delimited records, you will need something more than text. However, I still don't know of tools to work with json from the command line that are as concise, efficient, flexible and robust as the standard unix utilities for text.

> How do you do that with json?

Depends on the schema used, but it would probably be something like

    jshon -a -e "ip" -u < log | sort | uniq -c | sort -nr | head

The problem with your first command is the "cut" - works great for a while. Then you deal with other_vhosts_combined.log, and there's a leading hostname and port, so you want -f2 instead. After that, you start working with the user-agent field, and after remembering that the date and request fields have internal spaces, settle for something like -f13, which works until someone puts a space in their faked-up referer, and you can't use cut any more.

I love plain-text logs, but that apache logformat is a bad example of "neatly delimited records", given the unstable mixture of spaces between and within fields (some of which are quoted, some not, and one even quoted with square brackets!).

Not JSON, but one vision as to how structured data could fit into UNIX ecosystem: http://xml-coreutils.sourceforge.net/

In your current pipeline just replace `cut` with an XPath expression, everything else stays practically the same.

Of course current XML ecosystem has different tools that makes your case trivial, though that’s something new to learn.

recordStream provides some of these features. Take a look here https://github.com/benbernard/RecordStream

Reply on news.ycombinator.com