Jq: A lightweight and flexible command-line JSON processor | Hacker News Reader

Jq: A lightweight and flexible command-line JSON processor

stedolan.github.io

229 pointswtetzner55 comments

Open articleView on HN

jq is written in portable C, and it has zero runtime dependencies.

Thank you! When I clicked on the link I was fully expecting it to be written in Node and I was going to cry.

Why is having node as a dependency such a bad thing? No snark. Genuinely curious.

The subsequent sentence suggests that the benefit is convenience of impromptu deployment:

> You can download a single binary, scp it to a far away machine, and expect it to work.

I don't think that node is bad, but it's simply another dependency you have to worry about. Most devs likely won't have node pre-installed. Having a no-dependency binary is a great thing.

Node is great, and a port of jq to node would be awesome (someone once hinted at such a port, but I've lost track of it).

But jq also has libjq, a C library. And jq uses a copy-on-write, reference counted representation of JSON values, which is, for example, inherently thread-safe (though jq isn't using atomic operations for refcount management yet). The library is easy to use and powerful.

Ultimately the main thing I love about jq (and why I contribute to it) is the jq language itself. It reminds me of my one-time favorite, Icon. But in a world where C is still the champion of systems programming (until Rust takes over?), it's real handy to have a C JSON library _and_ a functional DSL that's easy to invoke from C.

One of jq's maintainers (nicowilliams) often mentions Icon as a source of inspiration.

@nicowilliams (github) and cryptonector (HN) are one and the same ;)

It's no secret either, just look at my commit messages' email address :)

Because if you are not developing in node, and don't have any other node based software, yet another runtime, its package manager(s), and dependencies are unwanted additions to a system.

They use this tool extensively at Spotify. I saw a presentation by https://twitter.com/jooon, when I was over there in September last year.

It is a really cool tool. Especially since they use JSON internally for almost all of their data representations.

Jq + Percol[1] is kind of cool.

[1] https://github.com/mooz/percol - was on HN a few days ago.

If percol is based on aut-fu, then it's probably not that good. The screenshot shows auto-fu, I know that zsh-plugin, but it's really messy and does very aggressive auto-completion (erasing your previous input). I'd rather use fish-shell instead.

no percol is not shell plugin, but a simple cmdline util used in a pipeline, I suggest you take a look. I understand that animation on the website is a little bit confusing though.

oh, cool! I'll take a look then. =)

I was also thinking of Xiki (http://xiki.org/) + jq.

A Unix-like shell with JSON for object serialization could approximate the PowerShell, no? Think of ksh93's awful compound variables... done right.

Is this the link to that presentation http://www.slideshare.net/jooon ?

This is a really neat tool.

If you just need pretty printing and don't have jq installed, you can use the python command line with the built in JSON module:

wget http://reddit.com/.json | python -mjson.tool

# (fixed) output to stdout:

wget -O- http://reddit.com/.json | python -mjson.tool

This script drops you into a coffescript REPL with the json loaded into memory under the variable j.

I find it nice to be able to loop the data, define new subsets, filter it with functions, etc.

It was a 10 minute slap-job to get it together, so it does barf on particularly large datasets. The obvious reason is that it serialises the JSON into an argument for a child process. Shouldn't be too hard to resolve if anyone feels like it.

https://github.com/davidbanham/dotfiles/commit/0ea950373604e...

I've been using this tool to build a bash library for a JSON/REST API and I can attest to its usefulness. Filter huge data sets down to just the fields you want, mutate, select, search, filter. Most everything I've wanted to do this can handle. Plus it's fast and dead simple to install. Kudos to the devs for this fantastic tool that I hope will soon be part of the default CL toolset across the board.

I'm using this tool with curl in scripts calling AWS api. Looking for an id of an object (security group, autoscaling group etc.) with its name and referencing it afterward in another command.

Hey, me too! It's pretty awesome, although the documentation/examples aren't as great as they could be - it took me some time to figure out how to properly parse arrays, and I'm still not sure how to actually do what I want to do (I want to convert an array of two instance objects into an array with just two instance IDs.

    % aws ec2 describe-instances --instance-ids '["i-3026a249","i-28739551"]' \
    | jq '[.Reservations[].Instances[].InstanceId]'
    [
      "i-28739551",
      "i-3026a249"
    ]

Thanks!

I've been using it for the exact same thing. One difficulty that I had was that you need to use the raw option, "jq -r" for string output without quotes, but I was looking at the query language documentation and not finding it.

EDIT: here is one I used. Given VOLUME_ID, gets the most recent snapshot-id.

    SNAPSHOT_ID=$(\
    aws ec2 describe-snapshots --owner-ids xxxxxxxxxxxx --output json | 
      jq -r "
        .Snapshots | sort_by(.StartTime) | reverse[] |
        select(.VolumeId==\"$VOLUME_ID\" and .State==\"completed\") |
        .SnapshotId
      ")

EDIT2: now that I'm looking at it, I think that I expected it to work with multiple snapshots but I'm not sure that I tested it.

I've been using this to transform the output of ElasticBeanstalk environment queries into things I can feed into environment updates since moving between versions is a PITA. Should post that code sometime...

Does anyone know of a similar tool for XML? In particular, it would be cool to have a useful way to run xpath, and maybe xquery, from the command line. The tools I've found have unfortunate interfaces, e.g. they won't read XML from stdin.

Have you tried XMLStarlet[1]? It's my go-to for command line XML processing.

[1] http://xmlstar.sourceforge.net/

Even with libxml's xmllint you can do some xpath stuff: http://wiki.apertium.org/wiki/Xml_grep

You might check out these:

http://www.w3.org/Tools/HTML-XML-utils/

However, I just use a Ruby script which loads the Nokogiri library:

https://github.com/clarkgrubb/data-tools/blob/master/src/dom...

https://github.com/clarkgrubb/data-tools/blob/master/doc/dom...

I'd just use xml2json and then apply jq. Usually gets the job done. The benefit of that is that you only need to learn jq.

See this blog post I wrote a while ago for an example: http://jeroenjanssens.com/2013/09/19/seven-command-line-tool...

Most of the JSON conversions from XML get it all wrong though. XML is not really an object (dict, hash) based meta-schema. It's an array-based meta-schema. XML->JSON conversions should look something like this:

    [ {"tag":"something", "attributes": { ... }, "nodes": [ ...] }, ... ]

with nodes being objects with one key to indicate if the node is a text node or an element, and if a text node then a value, and so on.

xmllint - http://xmlsoft.org/xmllint.html

I am not a huge fan of PHP, but you can do some useful stuff to XML using the PHP command line tool, provided you have the right libraries installed. I have run into situations where the package manager on a machine I'm using doesn't have an up to date xmlstartlet or xmllint, and have tidily gotten around this problem using PHP.

I've been using it for over a year - especially with AWS CLI, but, I'm a little displeased that it only has the query feature (duh) and does not have a little more like merging JSONs (I think it only has appending). Another issue is that I wasn't able to find release notes for v1.4 - what's new since v1.3? There other similar tools, but they require node, Python packages, etc. jq really is installed with one command and no dependencies.

You should be able to do some pipe fu and carry out some form of merging via bash. I mostly use it for making our output more readable or producing CSV style reports with the filtered data. Its filters are probably the biggest feature, at least in my mind.

Sorry for not being clear. I meant a merge in the form of {"a": 1, "b": 2, "c": 3} merged with {"b": 20} returning {"a": 1, "b": 20, "c": 3}. Like Python's update() or PHP's array_replace_recursive() although PHP's a little richer.

If you can code it using jq itself, you can submit a pull request and have it added to the builtins.

    jq 'reduce .[] as $obj ({}; .+$obj)'

    jq -s 'reduce .[] as $obj ({}; .+$obj)'

should do it. `+` adds objects by merging them.

Thanks! I'm eager to try this and hopefully clean up my code!

Yeah, I'm a schmuck, I forgot to add release notes :(

I've been thinking I need to do a 1.4.1 release just for that!

(Plus we're getting some cool new features suddenly. @wtlangford (on github) is working on regexps, for example. Maybe a 1.4.1 so soon would be justified.)

No, you're anything, but a schmuck! Thanks for the great project!

Been using jq for a while now. Probably since the last hn post. My biggest gripe with jq is the name and asking for help on stack overflow - the jq tag is an alias to the jquery tag. Someone with tag edit level rep should really fix that.

You can probably make a post on meta stack about it

http://meta.stackoverflow.com/search?q=jq

the system works!

Here's a cute little jq script I used to munge some json around.

https://gist.github.com/tbelaire/d00cb77fb0db7b38bed2

I just use it just as a pretty-printer for the console (jq '.') , and it does the job very well.

I can't stress how useful this is. I use it to process AWS SQS messages for debugging. Incredibly powerful tool.

I have also used it to validate sudokus, because I can.

https://github.com/slapresta/jq-sudoku

If you get and build latest from master, you get the --sort-keys commandline option

And just like that, you have an amazing base for diffing json documents of arbitrary complexity:

    vimdiff <( cat doc1 | jq --sort-keys . ) <( cat doc2 | jq --sort-keys . )

It's in 1.4!

I've been using this for a while, and it's wonderful to parse the output from API calls requested via curl, as well as quickly checking a specific package.json field from the npm output. The --json flag is handy for the latter.

Awesome stuff, thanks! Will give it a try really soon.

This can help Docker automation by processing the output of `docker inspect`.

This is a great tool. I've longed for an equivalent for YAML.

underscore-cli: a similar tool, in JS (node)

https://news.ycombinator.com/item?id=7896054

Cool, just grabbed it with sudo apt-get install jq

Reply on news.ycombinator.com