Convert JSON to a Unix-friendly line-based format

"Everyone I know prefers to work with JSON over XML, but sadly there is a sore lack of utilities of the quality or depth of html-xml-utils and XMLStarlet for actually processing JSON data in an automated fashion, short of writing an ad hoc processor in your favourite programming language."

Actually there is just such a suite of utilites! See https://github.com/benbernard/RecordStream

Maybe you're solving a different problem, but I'm not sure emitting one key-value pair per line is ultimately the way to go:

  $ echo '[{"a": [{"b": {"c": ["foo"]}}]}]' | jsonpipe
  /   []
  /0  {}
  /0/a        []
  /0/a/0      {}
  /0/a/0/b    {}
  /0/a/0/b/c  []
  /0/a/0/b/c/0        "foo"

In my own work, I've built up a suite of stream-based nested record processing tools that accept & produce JSON, protocol buffers, and a unix tab-delimited format. For the unix format it's been more useful to stick to the standard one-record-per-line thing, and let the user specify what fields to extract and their order.

Here's a depressing example of the fun you can have with new media and old unix tools, to give you some idea:

  $ mill io -r json -w texty -W fields=in_reply_to_screen_name < tweets.json | \
    grep -v -E '^(None|)$' | sort | uniq -c | sed -e 's#^ *##' | \
    sort -k1,1 -nr | head -5
  258 justinbieber
  248 ddlovato
  184 gypsyhearttour
  164 Logindaband
  145 Louis_Tomlinson

    import os.path as p

Renaming imports to single letters bugs me. I went to see where it's used, and it isn't. Binding pyflakes to Cmd+S in TextMate was the best thing I ever did, and it would have caught this. pyflakes is seriously awesome in that role, and pylint before committing.

I'm also surprised that simplejson is used, instead of the built-in json in Python 2.6 and up. A good solution is:

    try: import json
    except ImportError: import simplejson as json

Here is my contribution to Unix friendly JSON command line tools (requires Node.js and NPM): https://github.com/zpoley/json-command. Here are a couple other good ones: http://kmkeen.com/jshon/ https://github.com/micha/jsawk

Why not just use Rhino or any other stand alone javascript interpreter?

http://www.mozilla.org/rhino/

a simple example

echo 'x=[1,2,3];x[1]'|js

Line-based processing is still important! This work reminds me of an article I wrote 11 years ago covering Sean McGrath's work on PYX—a line-based format for XML—see http://www.xml.com/pub/a/2000/03/15/feature/index.html.

That work derived from that of Charles Goldfarb on SGML, dating from 1989 on ESIS, ISO 8879.

We'll always be downsampling to something we can use with sed, grep and awk. They're too handy not to.

> Because the path components are separated by / characters, an object key like "abc/def" would result in ambiguous output. jsonpipe will throw an error if this occurs in your input, so that you can recognize and handle the issue. To mitigate the problem, you can choose a different path separator:

Ugh. I should never ever have to pick details of the format to work around content in the format. The only real solution is escaping, though it does add complexity.

cat foo.json | python -m json.tool

Interesting. For faster flattening of nested structures into paths -- and the inverse operation, unflattening -- you could use this Python C extension:

https://github.com/acg/python-flattery

Full disclosure, I'm the author. ;) It uses "." as the path separator, but would be easy to allow "/".

The use case is a bit different, but I wrote a little converter to TSV, adding in a header. I mostly use it for input into R, but I use it a lot. It only works for fairly flat JSON objects. https://github.com/brendano/tsvutils/blob/master/json2tsv

I've written a "json" command line tool with nodejs that offers transformation of JSON with modern javascript expressions (with support for Array.map, Array.reduce etc).

http://fforw.de/post/scripting-json/