Jq – Work with JSON on the command line | Hacker News Reader

Jq – Work with JSON on the command line

200ok.ch

346 pointspreek80 comments

Open articleView on HN

It's always surprising when I learn that this isn't more widely known - it's such a core part of my daily workflow!

The links to the project aren't immediately obvious, so here they are:

https://stedolan.github.io/jq/ https://jqplay.org/

It's always surprising when I learn that people think jq is a rich tool.

https://metacpan.org/pod/App::RecordStream

I found, as a programmer, going a little less specific is more versatile. ojo[1], which I've compared to jq before[2], works well for this.

1: https://metacpan.org/pod/ojo

2: https://news.ycombinator.com/item?id=6419479

+1 for RecordStream

I remember using recs at Amazon. Is that where it originated ?

I don't know, but supposedly Amazon is somewhat hostile to publishing open source, so I doubt it.

I made a gui for it, if you're interested https://github.com/wellsjo/JSON-Splora

Just tried this! Really cool!

Is there any way that you could allow for the minify and format commands to be accessible from the in-GUI command-line?

Thanks :) https://github.com/wellsjo/JSON-Splora/issues/38

Amazing tool. Thanks!

Love jq and https://paw.cloud is another of my favorite tools for working with APIs, when curl | jq . Isn't enough.

An alternative that completely replaced jq for me: rq https://github.com/dflemstr/rq

Not limited to JSON, and super fast.

rq is super great, it's been indespsible for me while working with CBOR. It's also way more useful than jq if you want to filter down your data. Some examples:

    ~> curl -s http://httpbin.org/ip | rq
    {
      "origin": "184.100.112.11"
    }
    ~> curl -s http://httpbin.org/ip | rq 'at origin'
    "184.100.112.11"
    ~> curl -s http://httpbin.org/ip | rq 'at origin | split "." | spread | drop 3'
    "11"
    ~> curl -s http://httpbin.org/ip | rq -C
    ?foriginn184.100.112.11⏎

(These are a bit contrived, but that was the only public API that returns JSON I could think of.)

The only thing I haven't figured out is how to unquote string elements once filtered down to just that string.

I'm the author of rq, and it's great to hear about your use-cases!

Feel free to submit a feature request about the raw string use-case, I've been thinking about it for a while but some people have wanted me to add line support (maybe -l and -L?) and some people have wanted raw text support (maybe -r and -R?) while other people still have been talking about TSV/CSV support (not sure how that would look tbh...). I'm not sure what to do exactly, so feel free to tell me!

Aw dang, I wish I knew about this earlier. May have made teaching command-line data processing to beginners quite a bit easier. Example lesson using Spotify API (which is still public and free):

http://www.compciv.org/recipes/data/touring-the-spotify-api/

(just noticed that you used the Spotify API too for the demo :) https://github.com/dflemstr/rq/blob/master/doc/demo.md)

I personally like using jq but not sure how easy-to-grok it is for people who are new to both the command-line and to serialized data structures in general. Will have give rq a spin (waiting for it to compile on my machine) but it looks quite nice, at least for my own uses, which heavily involve YAML.

In terms of TSV/CSV support, you may want to checkout csvjson, which is part of the csvkit command-line suite: http://csvkit.readthedocs.io/en/0.9.1/scripts/csvjson.html

csvkit is my go-to tool for data, so much that I rarely ever use SQL for anything other than when I actually need a database. I suppose if rq were to be my go-to data-processing tool, I'd probably do this:

      csvjson mydata.csv | rq ...

Unless I'm missing it in the documents, there doesn't appear to be a way to convert to CSV? That would be helpful. FWIW, csvkit has a tool named in2csv, which will read from line-delimited JSON:

     curl -s 'https://api.spotify.com/v1/search?q=rihanna&type=track&limit=50' \
        | rq 'at "tracks.items" | spread | flatMap artists' \
        | in2csv -f ndjson

http://csvkit.readthedocs.io/en/stable/scripts/in2csv.html

A public JSON API intended for tests or demonstrations:

https://jsonplaceholder.typicode.com

I use jq daily and didn't know about rq, thanks!

Quick check whether it is super fast for a simple task like extracting a single field from 1M JSON objects:

$ time rq -j 'at "title"' < fixtures/1m.ldj > /dev/null real 2m16.692s

$ time jq .title < fixtures/1m.ldj > /dev/null real 0m26.040s

So jq is 8 times faster? Anyway, rq functions look useful, so will definitely check it out.

It requires v8 and does not compile on macOS successfully via Homebrew.

Ah, that explains why the executable is so large (14 MB, compared to 650 KB for jq).

That v8 crate looks interesting. There was a post a while back about Neon, for writing Node modules in Rust:

https://news.ycombinator.com/item?id=10786411

I don't have a Mac computer so it's hard for me to offer good Mac support. I'd really appreciate feedback on these kinds of issues so that I can fix them!

I _love_ jq. It's been an incredibly useful tool for me since I discovered it ~6 months ago. However, article mentions "jq -n"; I personally find jq syntax less appealing when it comes to generating JSON instead of parsing it. For that particular task, I prefer using "jo":

https://github.com/jpmens/jo

jq & jo are the Calvin & Hobbes of the shell: lots of fun together!

A few examples:

- Generate a simple JSON object:

  $ jo foo=12 bar=true baz=null
  {"foo":12,"bar":true,"baz":"null"}

- Want a nested object?

  $ jo foo=12 bar=true baz=null thing=$(jo number=57 test=string)
  {"foo":12,"bar":true,"baz":"null","thing":{"number":57,"test":"string"}}

- What about arrays?

  $ jo thing=$(jo number=57 test=string) anarray=$(jo -a 1 2 3 4 5)
  {"thing":{"number":57,"test":"string"},"anarray":[1,2,3,4,5]}

- Now let's add some jq magic to sum all the values in the array together:

  $ jo thing=$(jo number=57 test=string) anarray=$(jo -a 1 2 3 4 5) | jq '.anarray | add'
  15

Great stuff :)

Some prior submissions:

2 years ago:

https://news.ycombinator.com/item?id=9446980 (91 comments)

https://news.ycombinator.com/item?id=7895076 (55 comments)

4 years ago:

https://news.ycombinator.com/item?id=4679933 (36 comments)

https://news.ycombinator.com/item?id=4985250 (10 comments)

https://news.ycombinator.com/item?id=5734683 (29 comments)

I'm sure someone's going to throw a shoe at me for this, but...this sort of thing is why you want an object-oriented shell.

PS C:\> $x = ConvertFrom-Json -InputObject '[{"banana" : {"stem" : "brown", "peel" : "yellow", "flesh" : "white"}}]'

PS C:\> $x.banana.peel

yellow

I don't want an object oriented shell but I have on occasion found jq to be very useful.

$ echo '[{"banana" : {"stem" : "brown", "peel" : "yellow", "flesh" : "white"}}]' | jq '.[].banana.peel'

"yellow"

Come on now, you're missing the point. If the lingua franca of your shell is objects instead of text, you can import from whatever format you want and work with it in high fidelity:

PS C:\> $x = [XML]"<banana><stem>brown</stem><peel>yellow</peel><flesh>white</flesh></banana>"

PS C:\> $x.banana.peel

yellow

PS C:\> $x = ConvertFrom-Json -InputObject '[{"banana" : {"stem" : "brown", "peel" : "yellow", "flesh" : "white"}}]'

PS C:\> $x.banana.peel

yellow

PS C:\> $x = New-Object PSObject -Property @{banana = New-Object PSObject -Property @{stem = "brown"; peel = "yellow"; flesh = "white"}}

PS C:\> $x.banana.peel

yellow

PS C:\> $x = @{banana = @{stem = "brown"; peel = "yellow"; flesh = "white"}}

PS C:\> $x.banana.peel

yellow

Good point, thanks for clarifying.

I won't throw shoes. However, I do prefer anything that forces a commands output into text. Which JSON at least gets that right.

Specifically, if I have something that output objects, it is possible (and likely) that I can not persist that output to less and explore what is going on. Great if I already know exactly what I want. Terrible if I just want a visual exploration. Worse, if I can't dump it to file, I may have to constantly do some expensive operations just to find out what will give me the answer I want.

It might be interesting to try... but switching operating systems is kind of a big step just to try another shell.

Or is there some OO shell that works on Mac or Linux that I don't know about?

Python?

Hey guys, I work with JSON on a daily basis on machines that don't have access to jq. My advice to you in similar situations is as follows:

To pretty print JSON:

    cat myjson.json | python -m json.tool > prettyPrinted.json

To validate json:

    cat myjson.json | python -m json.tool
    echo $?

If the result is 0 it is well formed and 1 otherwise. You can also pretty print JSON by opening the scratchpad in firefox and clicking the "Pretty print" button.

Why do you have access to Python, but not jq?

I would guess: a policy against installing unapproved software on production servers, systems that don't have compilers installed (for security, jq appears to be C code), or debugging on client/customer machines

He might not administer (or have permission to modify) the machine, and Python ships with most distros.

If you use AWS CLI [0], you can embed JMESPath [1] queries via the --query switch, i.e. you don't ned jq. I was a loyal supporter of it, but switched to JMESPath [1], and I love its query languages more.

[0]: https://aws.amazon.com/cli/

[1]: http://jmespath.org/

JMESPath is nowhere near us powerful. No unique(), for example, which is very important when munging JSON

Okay, but in 99.9% of the cases it's powerful enough.

Azure also uses this in their new CLI [0]. The biggest advantage of JMESPath over jq is that JMESPath is a spec, and thus can be implemented in several languages [1].

[0]: https://github.com/Azure/azure-cli [1]: http://jmespath.org/libraries.html

My jq killer:

  function ruby_json {
    ruby -e "require 'json'; require 'active_support/all'; puts JSON.parse(STDIN.read)$1"
  }

Performs/prints arbitrary ruby over a readily available representation of the JSON object coming from stdin.

Example:

  curl the_endpoint | ruby_json "['funnels'].map{|a| puts a.to_i * 2 }"

Cannot accept user inputted script for json transform because Security nightmare! That's JQ's and JMESPath's secret sauce.

i want to love this tool but it's almost impossible to do anything complicated. the docs have a lot of examples but they sort of stop sort

I've been using jq quite a bit lately and agree that the docs would benefit from more complex examples. However, I've been able to get some fairly complex solutions done with jq. Do you have a concrete example of something you are trying to do?

I have server logs in jsonlines format in a file. My goal is to display some basic information (URL, referrer, time stamp, IP address, and user agent) for each log line with a particular header set to a specific value. Headers are in $.request.headers and are an array of key-value pair arrays.

Figuring out this use case from the jq docs or jqplay has been a major struggle for me. I feel that this use case reflects 90% of what I would want to use jq for. I feel that if I can't get over this hump just from docs, there's no way I could justify bringing this tool on board with my team.

The JQ stackoverflow community is VERY helpful.

Sounds like you're after the select() filter, which evaluates a boolean expression and passes the input to the output only if the expression is true.

So your example would be something like: select(.request.headers|map(.[0] == "X-My-Header" and .[1] == "my target value")|any)|<pick out some keys to display>

I realise though that your point wasn't the specific example, it's that the docs for these kind of cases are poor, and you aren't wrong.

I guess my main advice is to not think of jq as something like cut or sort - instead, it's more like awk or sed. You can do lots of crazy things with sed, but it's not immediately obvious how just from reading the man page. These kinds of tools require a little more time investment but are very powerful.

(edit for code typo)

jq is awesome, but what stands out to me in the article is the use of curl. You can shrink those ~200 bytes of curl by 2/3rds using HTTPie(https://httpie.org/):

  http put your.api.endpoint email=your@email.address password=swaggerrocks

HTTPie and jq go together like peanut butter and chocolate, except without the caloric bloat.

And swift death to those with allergies.

Hi, I wrote a similar tool: https://github.com/edmund-huber/jsonq

Looking at the jq github repo led me to also find ag, an ack-grep replacement. Both are in my distro's repositories.

https://github.com/stedolan/jq

https://github.com/ggreer/the_silver_searcher

"The command name is 33% shorter than ack!"

Skip ag and go straight to rg (ripgrep).

Now check out ripgrep (command `rg`) https://github.com/BurntSushi/ripgrep, the ag/ack killer ;).

I like ag, but unless I'm missing something, ack has this killer flag which ag lacks:

    --output=expr
       Output the evaluation of expr for each line (turns off text
       highlighting) If PATTERN matches more than once then a line is
       output for each non-overlapping match.  For more information please
       see the section "Examples of --output".

Basically, it allows you to use captured groups in the output:

      cat file | ag '(\w+) (\d+)' --output '$1 is $2 years old'

On the other hand, ag allows multiline matching. So I end up using ag and ack together frequently.

    sed -rn 's/([a-zA-Z]+) ([0-9]+)/\1 is \2 years old/p' file

sed doesn't support the same PCRE regex ack or ag

No, but the `-r` option makes it use extended regex[1]. Here's some PCRE:

    cat file | perl -ne 'print if s/(\w+) (\d+)/\1 is \2 years old/'

My only point is sometimes old school standard tools are plenty.

1: https://www.gnu.org/software/sed/manual/sed.html#Extended-re...

Right...but there's a point in having a tool that does PCRE, and all the things useful in grep, in one command, rather than kicking out to Perl.

Agreed, it's nice to consolidate disparate functionality into a single invocation.

But when I saw mVChr's sed line, and then again the perl line, I was reminded "Oh, right, old school." And I wondered for the Nth time how much of old school has been unknowingly reinvented, and how many times.

Nevertheless, I like the new(er) tools in this thread too.

ripgrep has that with -r/--replace.

Related HN discussion, but more encompassing of things you might need to do on the command line with data: Command-line tools for data science (2013) [1]

1: https://news.ycombinator.com/item?id=6412190

The Wishbone event stream processing framework (http://wishbone.readthedocs.io) has the wishbone.flow.jq module (https://github.com/smetj/wishbone-flow-jq) to do pattern matching on JSON streams. The article http://smetj.net/processing_webhooks_using_wishbone_part_2.h... shows how it's used.

Friend wrote a feedforward neural network using jq.

https://github.com/kevin-albert/jq-neural-network

Recently discovered jq and it has been indispensable part of my toolbox. A lot of json tasks I used to write scripts for, such as filtering and multi file concat, are now possible for a short command.

I wish jq was available from Java. We built a microservice recently that was just an aggregator of a bunch of other api calls. Jq would have been perfect.

jmespath is:

http://jmespath.org/libraries.html

(also, a nicer language IMO, even if it has slightly fewer features)

A few months ago I wrote a blog post about how to use jq in conjuction with pup to debug HTML-in-JSON endpoints: https://the1x.engineer/2016/08/20/efficient-endpoint-testing...

jq is one of my favourite tools - can't recommend it enough.

jq is awesome. While it can do a lot of really cool processing to filter out nested details, the majority of my usage of it is:

    $PROGRAM_THAT_OUTPUTS_JSON | jq .

Which simply pretty prints the JSON input with two spaces per indentation. Also, $PROGRAM_THAT_OUTPUTS_JSON is usually a script that simply outputs the clipboard (alias for "xclip -o").

My usage is generally pretty simple as well but if people are looking for a more complex example, I wrote one yesterday for putting together some test data. Turns DynamoDB responses into newline separated flat arrays:

    pbpaste | jq -c '.actionResponses[].data.items[] | [.attributes[] | .stringValue // (.numberValue | tonumber)]'

This is actually awesome for demos in lectures! Great to have better tooling for such a regular task on the command line.

Conceptually similar is Mike Bostock's wonderful ndjson-cli[0]

[0]: https://github.com/mbostock/ndjson-cli

jq is really nice and I always install it as part of my tool chain. However I don't use it very frequently because of which I can never remember the syntax which imho is not very intuitive :(

For shell scripting, jshon is also very useful ( http://kmkeen.com/jshon ).

I actually just discovered this tool a few days ago when I wanted to do something with the Gitlab API from a bash script. It's really cool!

Q? How does Jq handle crap (malformed) JSON? ... quickly scanned the docs, tutorial and readme. I see no FAQ.

For all of you on Macs, it's as easy as `brew install jq`. One of the first things I grab on a new box.

Seems like you could totally write a graphql command line utility to filter through json output.

This basically becomes a necessity when working with the AWS cli. Great tool!

The aws cli tool basically has this built in with --query

Anyone here used jsawk that could compare and contrast to jq?

jq embeds fp bits, you can define functions too, crazy.

Very nice!

Reply on news.ycombinator.com