Interactive examples for learning jq
ishan.page
ishan.page
My solution was literally to create Anki cards every time I discover a neat feature that I might not remember but it would be useful too. I just go through it once a day for 10 minutes (my anki cards) and it works like a charm. My memory for various cli tools has drastically improved. Rarely do I need to reach for Google, man docs or ChatGPT for most cli tools usages. I’d recommend spaces repition for cli tools
Happy to see others have been doing that too!
I will never truly memorize how to use this because it's not my primary goal, nor is it the end product to process data.
Rather, it is a means to a means to a means to an end.
jq lets you select elements like it's a JavaScript object using dot notation and array indexing.
jq '.key.subkey.subsubkey'
jq '.key[].subkey[2]'
jq '[ .[].key ]'
jq '{key1: .key1, key2: .key2}'
You put it all together into something like this:
curl https://api.github.com/repos/stedolan/jq/issues |
jq 'map({title: .title, labels: .labels}) |
map(select(.labels)) |
map({issue: .title}) |
sort_by(.issue) |
[{issues: .[]}]
Anyway, this article is neat! Good work!
If I were to nitpick one of the last examples with path has no explanation and flew over my head (would have to open the documentation), and a reset button for each example might be nice after messing with it a bit, but it was a nice play.
Regarding the reset button: I think that's a great suggestion and now it bugs me so much that I can't reset it. I'll add a reset button later tonight when I'm off work.
Regarding the confusing example: Yes, some of the examples are missing explanations (mainly because I spent more than a month on this post and I just did not want to put off putting it out any longer). Sorry haha. I'll try to improve the explanations and add more.
It really is a super cool little, super expressive nearly (if not entirely) turing complete pure functional programming language.
You can:
* Define your own functions and libraries of functions
* Do light statistics
* Drastically reshape JSON data
* Create data indexes as part of you JQ scripts and summarize things
* Take JSON data, mangle it into TSV and pipe into SQLite
cat data.json | jq '<expr>[]|@tsv' | sqlite3 -cmd ".mode tabs" -cmd ".import /dev/stdin YourTable"
As a part of the library you can compile your expressions down to "byte-code" once and reuse them.
Saying JQ is a best kept secret is an understatement. JQ gets more amazing the deeper you dig into it. Also it is kind of crazy fast for what it is.
edit: Formatting fixes
I’m sure at this point that many ETL jobs in notebooks we run at $BigCo today could be reduced to jq expressions that run 100x faster and use 1/10th the memory.
As soon as jq scripts reach a certain level of complexity I break out to writing a node script instead.
And given how rapidly jq scripts acquire complexity, that level is pretty low. One nested lookup, and I’m out.
Also the ‘nearly’ part is because I don’t remember if it has infinite loops or if it is more like Starlark and thus decidable. I do have vague recollections of causing infinite cycles in JQ, it quite as well could be entirely Turing complete.
So far I have not found a single task that JQ was incapable of. And I have abused it pretty bad on my spare time =], for intellectual challenge.
The brainfuck one is also gonna be going into my notes. That implementation is quite a terse implementation.
BTW your very welcome to help improve the jq documentation. Me and some other maintainers have been talking about that it probably needs an overhaul to be more approachable and also better document some nice hidden features. Join the discord if you want!
foo.bar.[0] = baz
foo.bar.[1] = bop
and then copying and pasting the results into jq makes the whole iteration loop much much tighter.
As for learning it, it's the same with any tool, key is repetition and regular use.
The tool for that is likely https://github.com/tomnomnom/gron It's probably the best tool to go back and forth easily between json and line oriented.
The first and foremost thing to know about jq is that it's built on path expressions, so the first thing to learn is how to write path expressions. Fortunately path expressions are easy in jq!
.a # Get the value of the "a" key
# in the current input object
.[0] # Get the value of the first
# element in the current input
# array
.a[0] # Get the value of the first
# element in the array at the
# key named "a" in the current
# input object.
#
# I.e., path expressions chain:
.a[0].b # Get the value of the "b"
# key in ...
Things get really interesting when you see that `select(conditional expression)` can be used in path expressions joined with `|`.
Just this can be very useful. It's also useful to know about the magic `path()` function, and `paths`, which I often use to just list all the paths in an input JSON text. Try applying `jq -c paths` to a `kubectl get -o json pods` command's output!
result=$(echo "$data" | jq -r '.optional // ""')
if [[ -n "$result" ]] ...
result=$(echo "$data" | jq -r '.optional')
if [[ "$result" != null ]] ...
And when using the raw-output option it helps with the ambiguity between "null" and null.
It doesn't use ?? because it predates that operator's introduction into the language by about 8 years
I find the biggest problem with jq is that the feedback loop is not tight enough. With this jq-repl the expression is evaluated at every keystroke.
Let me piggyback to mention the (neo)vim plugin I use for tightening the loop... https://github.com/phelipetls/vim-jqplay
It's great for building large complex queries that will eventually live in scripts, but your zsh plugin seems to hit a real sweet spot of fast feedback for ad-hoc queries too! Huge props!
zshbuiltins(1): Unlike parameter assignment statements, typeset's exit status on an assignemt that involves a command substitution does not reflect the exit status of the command substitution. Therefore, to test for an error in a command substitution, separate the declaration of the parameter from its initialization.
local query="$(__get_query)"
local ret=$?
local query; query="$(__get_query)"
The way I did this was to store both the last working query and the last working output, I'd only reuse it if the last working query was a prefix of the current query - that avoids the awkward case where you are deleting letters from the output, so you need an output further back in history (which I didn't store, wasn't worth the hassle)
Feature request?
To than end I wrote a line of jq to emit every structural path from any json as a list of jq arguments.
You can use it to make queries or keep track of a documents structure.
I continually bounce off the "language/philospohy" of jq in quite embarrassing ways. Every time I go "Ah, I can use this as a reason to learn jq and half an hour lateI've written a python script to extract the data instead.
https://zwischenzugs.com/2023/06/27/learn-jq-the-hard-way-pa...
JQ really is the best kept secret in data.
One thing I noticed, and where I stopped continuing, is that the jump from Filtering Nested Arrays to Flattening Nested JSON Objects, is WAAAY too big. From a simple filter to triple nested filters with keywords that had no introduction in a simpler example, isn’t working for me
> Aioli is a library for running genomics command-line tools in the browser using WebAssembly. See Who uses biowasm for example use cases.
Out of any language I’ve worked with, Ruby was #1 overall in being able to solve problems closest to how I thought about them conceptually if that makes sense. In other words, getting some way of doing something out of my brain and translating it to Ruby was the most seamless (I haven’t tried any real lisp in earnest), with ES6 a pretty close second now.
Even though I use it a lot professionally, I really don’t like Python much, and I like Java / Gradle less. The whole “There should be one-- and preferably only one --obvious way to do it” thing never held water in my opinion, and things like making an HTTP request then doing something useful with the output are really not fun in Python without using things like the Requests lib… even closing files or having to care where a cursor is, like what year is it again? Python doesn’t feel like a high-level language many times since it keeps forcing me to deal with minutia and doing the wrong thing by default. Then there’s the whole disappointment with PEP 582 getting rejected (check out the PDM project to see what Python packaging could have been) and I just can’t help but really despise it despite how useful it continues to be.
Sorry, just hearing the desire for anything else to be more pythonic in any way takes my brain to a dark place of PyTSD.
Hey, at least we don’t have to write TS types for JQ (yet).
I do a quick ctrl+r, type jq, and I can find all of my JQ snippets I've used in the past couple of years. If I then type "select" I can find all of the times I've used that function, etc.
I also use it to find while loops, kubectl snippets, environment variables I exported to run a script, etc.
1. JSON: `curl` to get interesting JSON APIs, `gron` and `grep` to explore what's inside them, `jq` to process them into interesting formats.
2. CSV: Lots of good choices here. `xsv` is very popular but I think development ended a while back; I like the `csvkit` just because I like tabbing through the options you have here. `miller` I've heard good things about. Or go to the total opposite end of the direction, use Simon Willison's excellent `csvs-to-sqlite` in conjunction with `datasette`, and then do a foray into the many interesting things you can do in SQL.
3. Bespoke text formats - `sed`, `awk`, and possibly even Vim macros reign supreme here, along with the rest of the "standard" Unix text kit. The big benefits of introducing these last is that these tools work as a superset of many of the previous ones for added flexibility.
Python (dead simple, easy to recall):
if monkey == 'fat':
print('happy')
if [ "$monkey" = 'fat' ]; then
echo 'happy'
you gonna forget about the semicolon, the spaces around the condition and it will error out. not to mention integer comparison operators...
python also has some terrible syntax, but those are advanced things, like list comprehensions.
jq...it's the same as awk, sed, bash... hard to remember for the reasons mentioned above
You know, a list of reserved words, what their functions are, how the structure works, etc ... the kind you'll see if you pick up some "intro to <language X>" book from no-starch or o'reilly or the kind that GNU Awk/sed/dc have.
If anything, it’s _too_ formal and may be missing some example for more complex usages that combine several features.
Interestingly I still remember most Perl5 syntax, even the crazy stuff, quite vividly, after some 6-7 years of not-writing Perl code. I wonder why - perhaps because Perl is not so complex (even the PCRE), and perhaps because one needs jq now and then, while Perl can be a primary tool for many things. Sadly, Perl is past its prime now, and there are no implications it'll ever do a comeback.
On an empty command, the context is the top-level of your JSON. As you add filter stages, that context evolves.
(this really requires more explanation and diagrams than I have room for in this margin)
The same thing happened to HCL.
I looked at aoili briefly. I didn't see how to reproduce the build.
(and regexes)
yaml2json | jq ...it's great for converting nearly any format to json for querying or transforming with jq
Carting your request over to chatgpt rather than learning the basics of jq doesn't make much sense.
I basically know enough jq to traverse a document, which took a few days of muscle memory. Totally worth it. One of the best tools I've used in the last decade and I'm barely scratching the surface.