Hacker News Reader

Comment by memset | Hacker News Reader

I personally have found a lot of difficulty with JSON-based logs, because tools like `grep` and `cut` don't suffice; I usually have to write a more complex regex with `sed` in order to extract useful information I want.

Anyone have a good utility for managing JSON in log files?

View on HN

RecordStream is a great solution for manipulating JSON log files: https://github.com/benbernard/RecordStream - makes it easy to perform relational-style operations, such as grouping, counting (recs-collate), and even joins (recs-join). It can also display and transform the logs in a variety of ways (recs-tocsv, recs-totable, recs-tognuplot for graphing).

There are also tools to extract records from other data formats such as CSV or by querying a database.

All in all, a fantastic tool for manipulating and querying complex data from the command line. I use it regularly.

Moving away from using regex to parse JSON, you can use Jsawk and Underscore-cli to manipulate JSON like Awk or JavaScript.

https://github.com/micha/jsawk/ https://github.com/ddopson/underscore-cli

But the problem is speed. I found that if parsing logs is slow, bug hunting is far less efficient.

underscore-cli is utterly brilliant for this, even better — dare I say it? — than traditional log files with cut/grep/awk. Being able to treat logs as a dataset that can be mapped/reduced/flattened/grouped/otherwise analysed is life-changingly good.

About the strongest endorsement I can give is that it makes me wish more of my logs were in JSON format, rather than wishing that less were as I always had before.

Recs (therecordstream.com) is great for this. A JSON log file would be interpreted as its native record format, so you can use all of its processing tools (like recs-grep and recs-collate and recs-sort).

Here's a bash JSON parser.

https://github.com/dominictarr/JSON.sh

Reply on news.ycombinator.com