JSON on the Command Line with Jq | Hacker News Reader

JSON on the Command Line with Jq

shapeshed.com

117 pointsgoranmoomin38 comments

Every time I try to do something involved with jq (i.e. more than a couple commands deep) and inevitably end up wandering through the manual, I get a persistent feeling that jq could be cleaned up quite a bit. That it either has structure throughout that I fail to grasp, or that it's really rather disorganized and convoluted in places. This is exacerbated by me never being able to find anything in the manual without skimming half of it each single time.

E.g.:

Filtering and transforming objects by reaching a few levels deeper in them seems to be way easier to slap together in Python's list comprehensions or some functional notation. Stuff like [{ * * x, qwe: x.qwe*2} for x in a.b if x.y.z == 1].

Add nested lists to the above, and I can spend another fifteen minutes writing a one-liner (i.e. `a` is a list of objects with lists in some field).

I once had to deal with a conditional structure where a field might be one object or a list of them. Hoo boy.

Every time I want to check the number of entries my filters print out, I need to wrap them in an array. Repeat this a couple dozen times during writing a complex query. Weirdly, this is where strict structure gets in the way―and while I understand the reasoning, I feel that something could be done for such basic need. (Like, aggregate filters should be able to aggregate instead of acting on individual items? Maybe it's already in the language, but I'm not in the mood to go over the manual again.)

Variables seem to be bolted on as an afterthought, or at least the syntax doesn't exactly accommodate them. Meanwhile, they're necessary to implement some filters omitted in the language. Compare that to Lisp's simple `(let)`. IIRC the manual also says that jq has some semblance of functions, but I'm afraid to think about them with this syntax.

I like the idea a lot, but execution not so much. Frankly I'll probably end up throwing together a script in Lumo or something, that will accept Lisp expressions and feed JSON structure to them. (I'd use Fennel, but JSON has actual null while Lua... doesn't.)

Btw, I have pretty much the same sentiment about Git. Git structures, great. Git tools, oy vey. Maybe I need Lisp for Git, or at least Python for Git.

> I like the idea a lot, but execution not so much. Frankly I'll probably end up throwing together a script in Lumo or something, that will accept Lisp expressions and feed JSON structure to them. (I'd use Fennel, but JSON has actual null while Lua... doesn't.)

This may come off as a little snarky but, you should totally give this a try and post the result on HN.

I say this because I both love jq for its functionality, but also have deep frustrations with wrangling it. I really encourage any attempts to improve it.

There's definitely room for a better jq - I would 100% try a version of it that was as simple to install, use, and just had a slightly easier to use language.

It would be very easy to do in Lua/Fennel, afaik. The problem is that Lua is so trigger-happy about `nil` that it doesn't distinguish nil's presence from absence, which won't do for JSON.

Now, I could attempt doing this in ClojureScript, but I'll get derailed to thinking about ClojureScript's inseparable conjoinery to Clojure and JVM, even if manifested in Lumo only as inexplicable boilerplate, and that gives me depression. Also Lua's start-up is faster than even bare Node.

Hy is probably the optimal choice. There's not even much to write: I can as well just spin up Hy's repl and read JSON in it. Maybe will need a few helper functions to save me from verbosity.

(Still) too damn complicated, yes.

I use it 1-2 times a month and always have to RTFM.

Yeah! I'm constantly RTFM for stuff that feels like it ought to be rather basic! That said, I love jq.

I agree. jq makes the easy stuff easy, but the syntax gets weird pretty quickly.

You might like fx [0]. Run on its own, it’s a command line JSON browser (you can even click on items to expand/collapse them), and you can also write little JS one-liners to do stuff like map over an array and transform the items.

Something like this, to extract the name from an array of users, for instance:

fx users.json ‘u => u.name’

0: https://github.com/antonmedv/fx

Personally I prefer https://github.com/jmespath/jp - it has a bit saner syntax based on an actual standard (at least for me it's easier to remember than jq).

And perhaps more importantly it doesn't mess with your numbers! Some other comment mentioned doubles, but I had a problem with Longs, jq would silently truncate them when they became to large to fit inside a float (I was working with random Long numbers used as secrets in some schema so it happened quite quickly in my use case).

Thanks for this; for the few cases/year I need something like `jq`, I just find its selector syntax too hard to remember (seems a common theme). `jp` seems way more usable.

Hey, just wanted to say that this comment made me create jql, an alternative to jq with a lispy syntax: https://github.com/cube2222/jql

Thanks for the inspiration!

Thread from 2016: https://news.ycombinator.com/item?id=13090604

2015: https://news.ycombinator.com/item?id=9446980

2014: https://news.ycombinator.com/item?id=7895076

2013: https://news.ycombinator.com/item?id=5734683

2012: https://news.ycombinator.com/item?id=4985250

https://news.ycombinator.com/item?id=4679933

(for the curious)

For those interested in JSON as a command line protocol, I think it's interesting to see how Andreas Kling has integrated it into his Serenity OS project. While the project is aesthetically retro, its architecture is forward-facing, and the usage of JSON for the representation of such data as the process tree is, I think, an interesting take on old Unix ideas.

It definitely makes writing the userspace code a lot nicer!

The main issue I've run into so far is that my JSON parser can generate quite heavy in-memory representations of parsed data. It's far from optimized though, and a lot can be done before needing to worry about the limitations of the format itself. :)

jq is the AWK of today. I'm in favour of jq in *nix base. I use it almost as much as grep | cut | sort.

JSON is ubiquitous now. So much so that even my <conservative enterprise customers>, who 20 years preferred data in CSV, 10 years ago preferred XML, now want to exchange data in JSON.

CSV still seems to be the standard in finance.

Precisely, because JSON is ubiquitous (unlike XML) and relatively safe to parse (unlike CSV).

It's not perfect - it doesn't have higher-order features like types/schemas or functions built in. But JSON + jq is a workhorse. I agree that distros do people a disservice by not bundling jq in base (and attempting to provide JSON output for distro-specific tooling).

Something better than jq please. It has the most arcane and inconsistent syntax that I have ever seen in a CLI tool. There are newer tools like jp with simpler language that can be understood by mere mortals with limited brain power.

You should check out GROQ (https://groq.dev), it's a query language for JSON.

A line of `jq` to turn any .json into a set of xpaths.

https://github.com/TomConlin/json2xpath

This is useful! Thank you for sharing!!!!

Jq can do lots of things that are similar to what’s normally done with awk, cut, sed, and grep. The languages pipeline model is slightly opaque (eg as soon as you use a function like select) and I feel tricky to get the hang of but overall I think it works well for the kind of job jq does.

That said, I usually get on well with a halfway house of piping jq -c ... into eg grep or sort -u for easy processing (-c writes one json object per line, aka compact form), or selecting a few elements, putting them in an array, jq-piping into @csv, and passing -r to get raw output which I can then process with awk or sort or other Unix commands.

One subtle caveat is that jq normalises all the json it processes, so for example escaped characters in strings may change and numbers which are not IEEE doubles get converted (eg if someone writes a 64-bit int or a bignum as a number not a string in the json)

Jq is nice and I regularly use it.

Does anyone know a more interactive json tool? With, for example, auto-completion? Showing possible fields to select next and maybe even possible values if fields are choosen.

> Does anyone know a more interactive json tool? With, for example, auto-completion?

https://github.com/antonmedv/fx

It just uses JS for the query language and can be hooked in to bash completion.

Personally I don't get on with jq, because I don't feel that plinking with JSON is something that warrants memorising a bespoke language.

I have been very happy with https://github.com/fiatjaf/jiq

I like https://github.com/raine/ramda-cli

It has interactive mode which gives you instant feedback. I also don't need to learn a completely new API, as I'm already familiar with ramda.

Being reminded of JQ has me thinking... I actually don't hate powershell or at least I like it's attempt at typed pipes.

Has anyone gone down the road of posix like Json pipes? Or maybe just Json bridging the gap between fully typed pipes and the string based Unix cli world?

Even just having a Json output format for the core posix apps would be good. If it had OS support, maybe you wouldn't even have to serialize to strings if both sides of the pipe happened to be compatible.

Yes, Oil Shell is one:

https://www.oilshell.org/preview/doc/json.html

Personally Id like to go a step even further. Id like to see the kernel provide some sort of "stddata" descriptor to build robust pipelines like powershell.

I like jq, but the documentation is super thin. Every time I go to do anything involved, I inevitably end up at stack overflow hoping somebody else has had to do it before.

I really wish there were a much greater selection of usage examples in the manual.

Jq is a great tool, I wish more people knew about it!

It's a bit arcane to use, but has proven itself over and over when doing little tasks with AWS CLI returns.

Exactly right.

I use jq to mangle get/list requests from AWS (and others) and then recursively hit each element for whatever reason. Without jq none of my workflows in this area would be remotely easy.

TLDR: Learn JQ! It is like sed and grep, but for JSON.

100%

If you're using AWS you're better using the built in option available for the cli. The --query option (and often combined with the --output option) does this, and uses JMESPath. Especially when you're scripting, it eliminates the need for having yet another tool.

That's a great point. Leverage --query for good justice.

Unless you need some additional juice, where jq supports actually transforming the return.

Quite arguably a heavier tool, e.g. a boto3 script becomes appropriate where --query falls short.

But options are a great thing to have.

> If you're using AWS you're better using the built in option available for the cli. The --query option (and often combined with the --output option) does this, and uses JMESPath.

Jq is so much more capable than jmespath that I find myself using jq on the json output of AWS CLI commands more than using the built-in query option. Sure, it requires another tool, but it’a often the best tool for the job.

I’ve been using it as a pretty-printer for a while now, it never occurred to me that you could use it to filter items out. In fact, I’ve been piping jq to grep to find things!

I'd like to have a command line tool that allows me to modify the json with javascript. Does jq provide any feature to process json that it's very difficult/cumbersome to do in js?

Jq allows you to fully transform a json structure. Batteries included. Similar to awk/sed etc you can even write your jq code in a separate file, and point jq to it.

Reply on news.ycombinator.com