An Introduction to JQ

One small but really convenient tip missing from the article is object shortcuts - these two commands are the same:

    curl -s https://api.github.com/repos/stedolan/jq/issues?per_page=2 | jq '[ .[] | { title: .title, number: .number } ]'
    curl -s https://api.github.com/repos/stedolan/jq/issues?per_page=2 | jq '[ .[] | { title, number } ]'

For anyone wrangling with data like I do. I use https://github.com/TomWright/dasel quite a lot (it supports various formats and conversion between them) Also csvkit https://csvkit.readthedocs.io for CSV to sql. And ofcourse pandas for analysis.

I love these - I use jq daily and am always happy to read about it from another angle.

The author probably has internalized more of the manual than they realize, and maybe improved at least one explanation; FTA:

> map(...) let’s you unwrap an array, apply a filter and then rewrap the results back into an array. You can think of it as a shorthand for [ .[] | ... ] and it comes up quite a bit in my experience, so it’s worth it committing to memory.

From https://stedolan.github.io/jq/manual/#map(x),map_values(x) :

> map(x) is equivalent to [.[] | x]. In fact, this is how it's defined. Similarly, map_values(x) is defined as .[] |= x.

Note here the casual introduction of the update assignment operator, '|='

This is a terrific introduction to a tool that I use often enough to pretend to be familiar with, but honestly just grep my bash history every time I need to use.

But I'd really like to see a discussion about the tool that the host website promotes: Earthy. It a build system, which is a family of tools that I've always hated, but it seems to be pretty decent. Is anybody using it?

I'm off to find HN threads on Earthly.

This is an amazing article. I think the two things that set it apart is (a) showing the process of building a complex command from scratch step by step, and (b) actually using a real API endpoint to start with.

One convenient tip I discovered after reading this article and trying out the command is that

    jq 'map({ title: .title, number: .number, labels: .labels | length }) | map(select(.labels > 0))'

can be refactored into

    jq 'map({ title: .title, number: .number, labels: .labels | length } | select(.labels > 0))'

or in other words, map(filter1) | map(filter2) == map(filter1 | filter2).

jq is unsurprisingly Turing complete, so I wrote a Whitespace interpreter[0] in jq. It is able to handle real-time I/O by requesting lines on-demand from stdin, which is the main input source, with `input` and outputting strings in a stream.

With a relatively large jq program like that, it is critical that the main recursive loop run efficiently, so it's annoying that there's no way to detect whether tail call optimization was applied, other than benchmarking. It would also be nice if object values were lazily evaluated so that it would be possible to create ad hoc switches.

[0]: https://github.com/andrewarchi/wsjq

The jq documentation is good, and thorough, but we see a lot of articles like these, and personally I think I would remember the particulars better (and maybe guess them without even referring to the reference manual) if the author wrote a narrative account of how they conceived of jq and how the language emerged from that conception. The tutorial is a great start, but something a little more fleshed out and introspective would help a lot, in my opinion.

As much as I love jq and it has helped me, it always has an „awk“ vibe to me which is one of the tools I use less despite its capability, because of the same reason: I tend to forget all the special syntax over the years.

Compare that with „gron“ ( https://github.com/Deitar13/gron ), which is arguably not as powerful and clunky, but it allows me to compose and tie into the other unix tools way better.

It‘s mainly for that reason I use it more than jq these days for ad-hoc analysis.

Good writeup!

One nit: ‘jq -r’ seems like more fundamental than a sidenote, especially considering it’s a cli tool. That could just be how I use it though (as glue between json and bash).

Another good tool in that family is ‘jtbl’. It provides table output, which is useful for cut, awk, sed and column.

Something I just learned about the other day was jid [0] to help query the json keys

[0] https://github.com/fiatjaf/jiq

For anyone struggling with jq's syntax, I recommend taking a look at fx [1] which uses JavaScript as the query language.

The author of the tool has also written a guide [2] and recorded a screencast [3] about the tool.

[1] https://github.com/antonmedv/fx

[2] https://medium.com/@antonmedv/discover-how-to-use-fx-effecti...

[3] https://youtu.be/ktfeRxKog98

> However, some things never stick in my head, nor my fingers, and I have to google them every time. jq is one of these.

However powerful jq may be, the comment above, which has exactly been my experience with jq, summarizes quite nicely the biggest hurdle with this tool.

99% of my jq use looks like this:

     cat file.json | jq . | <regular list of unix filters>

and the remaining 1% is straight cut and paste from google / stackoverflow that may or may not end up doing what I want.

jq's DSL is inscrutable

If you work with JSON and CSV data regularly, I'd also recommend checking out Miller: https://github.com/johnkerl/miller

From the project's description: Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

A very useful function in jq is "join", which I use a lot to cook the final shape of the data (many times used with fzf/dmenu)

Here's a simple way to list and browse github issues of a given user/repo:

    #!/usr/bin/env sh

    browse_url() {
      firefox http://github.com/$1/issues/$2
    }

    issue=$(curl https://api.github.com/repos/$1/issues |
          jq -r 'map([(.number|tostring), .title] | join(" | ")) | join("\n")' |
          dmenu -i -l 10 |
          awk "{print \$1}")

    browse_url $1 $issue

Although the `| join("\n")` part could be done in a more idomatic way with just `[]`, sometimes the manual way are still clearer to me:

    map([(.number|tostring), .title] | join(" | "))[]

My OpenAPI spec is 60,000 lines of machine generated goodness. JQ is great for doing non-trivial but useful things. Eg: Which endpoints are available if you have a specific OAuth Scope?

Most useful jq cli flag is -f. - take the jq script from file

Most useful tutorial for learning to manipulated your OpenAPI spec https://apihandyman.io/api-toolbox-jq-and-openapi-part-1-usi...

Big fan of JQ. I like it more than the traditional UNIX suite of text manipulation commands, because I get closer to "querying" rather than just filtering. It has really made me rethink where I want "interacting with a computer" to go in the future -- less typing commands, more querying stuff.

I have a few utilities involving JQ that I wrote.

For structured logs, I have jlog. Pipe JSON structured logs into it, and it pretty-prints the logs. For example, time zones are converted to your local time, if you choose; or you can make the timestamps relative to each other, or now. It includes jq so that you can select relevant log lines, delete spammy fields, join fields together, etc. Basically, every time you run it, you get the logs YOU want to look at. https://github.com/jrockway/json-logs. Not to oversell it, but this is one of the few pieces of software I've written that passes the toothbrush test -- I use it twice a day, every day. All the documentation is in --help; I should really paste that into the Github readme.

I am also a big fan of using JQ on Kubernetes objects. I know what I'm looking for, and it's often not in the default table view that kubectl prints. I integrated JQ into a kubectl extension, to save you "-o json | jq" and having to pick apart the v1.List that kubectl marshals objects into. https://github.com/jrockway/kubectl-jq. That one actually has documentation, but there is a fatal flaw -- it doesn't integrate with kubectl tab completion (limitation of k8s.io/cli-runtime), so it's not too good unless you already have a target in mind, or you're targeting everything of a particular resource type. This afternoon I wanted to see the image tag of every pod that wasn't terminated (some old Job runs exist in the namespace), and that's easy to do with JQ: `kubectl jq pods 'select(.status.containerStatuses[].state.terminated == null) | .spec.containers[].image'`. I have no idea how you'd do such a thing without JQ, probably just `kubectl describe pods | grep something` and do the filtering in your head. (The recipes in the kubectl-jq documentation are pretty useful. One time I had a Kubernetes secret that had a key set to a (base64-encoded) JSON file containing a base64-encoded piece of data I wanted. Easy to fix with jq; `.data.THING | @base64d | fromjson | .actualValue | @base64d`.

JQ is something I definitely can't live without. But I will admit to sometimes preprocessing the input with grep, `select(.key|test("regex"))` is awfully verbose compared to "grep regex" ;)

Very useful tool indeed. Worth investing some time in if you like doing data processing on the command line.

Stuff I do with it:

- prepare json request bodies for curl commands by constructing json objects using environment variables

- grab content from a deeply nested json structure for usage in a script

- extract csv from json

- pretty print json or ndjson output curl ... |jq -C '' | less -r. I actually have an alias set up for that.

The syntax is a bit hard to deal with. I find myself copy pasting from stack overflow a lot when I know it can do a particular thing but just can't figure out how to do it.

If there's an API for working with data that you know well already, it's somewhat pointless to learn the (quite esoteric) jq one.

Just pipe curl to node (https://github.com/jareware/howto/blob/master/Replacing%20jq...) or Python or Ruby or whatever you already know!

JQ is great once you get the hang of it. Some time back I had to write some simple tests on JSON output and I wrote a couple of helpers if anyone else is interested [0].

[0] https://gist.github.com/Checksum/72d927471c76c76c46418b3ee88...

I really want to like JQ. I know the tool is super powerful. Unfortunately I find the syntax obtuse and very hard to remember, especially when I only use it on rare occasions.

Everytime I want to use it I end up searching up examples of the syntax and not quite getting it right.

For exploratory purposes one can also use `jq 'keys'`

``` cat file.json | jq 'keys' | grep ependencies ```

This will list the keys, sometimes is really helpful with a big json that you don't know the schema, but you have the intuition that some key should be there.

I don't work with data, so my most common usage for jq is checking package.json properties from the command line:

  cat node_modules/some_lib/package.json | jq '.version'

i have a jq tutorial, where you can click on each step of the pipeline to view what it is doing [1], even had it featured here on hn, a while ago. [2]

[1] https://mosermichael.github.io/jq-illustrated/dir/content.ht... [2] https://news.ycombinator.com/item?id=22626080

Since the language is based on JavaScript, you can also concatenate strings together with “”+”” as part of your output, eg ‘.Name + “(“ + .Email + “)”

Most of the APIs in normal workflow are hidden behind a token. Is there a way to smooth the workflow for putting tokens in curl command ?

Excellent introduction!

It can really be overwhelming when you realize they control all of the major institutions in our country.

Has anyone figured out a workflow to nicely integrate jq filters into a postman type application?

Been looking for such a write up. Kudos to the author.

alias json_query=jq

no need to memorize

That title though.

Can anyone explain why I'm seeing JQ all over the web?! Yesterday a vendor gave a presentation saying that we should use it when using their command line tools, what's going on?