Hacker News Reader

Comment by anarcat | Hacker News Reader

I know I might be a dissenting opinion here, but I can never wrap my head around `jq`. I can manage `jq .`, `jq .foo` and `jq -r`, but beyond that, the DSL is just opaque to me.

And every time I try to learn, I get lost in a maze of a manpage:

https://manpages.debian.org/jq

I mean there are useful tips in there, but I rarely find what I need. I usually find it simpler to write a Python script (which ships with `json` because it's "batteries included") and operate on lists and dicts the normal way...

It's longer, but at least I don't need to learn a new programming language (if we can call jq that...)

View on HN

> the DSL is just opaque to me.

May I suggest "gron"? As compared to jq it is simpler (good!) has less features (good!) has no DSL (good!) and it is less powerful (good!, in many cases). It is a tool that "expands" json into standalone lines, and you can further process them using the standard tools grep, sed, cut, sort, awk...

EDIT: for example, if you have this json:

    {
      "outdir" : "out3",
      "data" : [
        {"img" : "img_01.jpg"},
        {"img" : "img_02.jpg"}
        ],
      "roi" : {
          "x" : 150,
          "y" : 150,
          "w" : 700,
          "h" : 700
      },
      "margin_h": 20,
      "margin_v": 5,
      "tile_size" : 300,
      "resolution": 0.5
    }

running it through gron produces this

    outdir = "out3"
    data[0].img = "img_01.jpg"
    data[1].img = "img_02.jpg"
    roi.x = 150
    roi.y = 150
    roi.w = 700
    roi.h = 700
    margin_h = 20
    margin_v = 5
    tile_size = 300
    resolution = 0.5

that some people find really convenient to deal with.

https://github.com/tomnomnom/gron

This online tool also produces assignment statements from JSON: https://www.convertjson.com/json-path-list.htm (note: creator here)

> I know I might be a dissenting opinion here, but I can never wrap my head around `jq`.

I don't think this is uncommon. People generally reach for jq for simple path expression evaluation against JSON objects, and never get deeper into it than that. It seems like the kind of thing you could re-implement yourself in an afternoon. However, as soon as you start taking advantage of some of the more complex functionality--say, a program like `.entries[] | select(.size > 1024) | .name`--there's a disquieting feeling of "what the hell is actually happening here?"

I took some time a while back to really dig into how jq programs work, and was surprised to discover how deep and powerful jq really is, while being built out of some very simple fundamental building blocks. Most of the jq builtin functions are actually implemented in jq itself[1]; even more of them could be, but aren't for the sake of performance. And the power and self-consistency of jq made sense when I found out that the creator Stephen Dolan is an "actual" Computer Scientist (in the academic sense) with extensive programming language research experience[2].

The main obstacle I see for most people is the lack of of an accessible introduction to how jq actually works when it comes to streams, function calls, generators, and backtracking. They're explained somewhat in the docs and on the project wiki[3][4][5], but in a fairly blunt way that assumes existing familiarity with terminology and concepts. It takes some effort to learn initially, but once you understand it, everything falls into place. I'm hopeful that subsequent installments in this "mastering" series can explain how the jq model works in an approachable way.

[1] https://github.com/stedolan/jq/blob/master/src/builtin.jq

[2] http://stedolan.net/research/

[3] https://stedolan.github.io/jq/manual/#Advancedfeatures

[4] https://github.com/stedolan/jq/wiki/Advanced-Topics

[5] https://github.com/stedolan/jq/wiki/Internals:-backtracking

Jq is really powerful but I think the bit it gets wrong is in array handling and filtering arrays. It is such a common thing that some syntactic sugar would be justified, but jq requires you to reach for map() and select() to do even the simplest stuff, and I can never remember if I should filter the `.array` or `.array[]`.

Yq [1] syntax for that is much better, and yq can accept JSON files too.

[1] https://github.com/mikefarah/yq

Yeah, the fact that jq has both 'streams of values' and 'arrays of values', and that the two are somehow completely orthogonal and support different operations is a huge design wart, in my opinion.

Let's just say it: jq is an amazing tool, but the DSL is just bad.

This is not to shame the developers but to voice our issues with it since this seems to be the prevailing sentiment, perhaps some brave soul will try and simplify it for human brains.

I cannot remember the syntax and have to search for it every time but it's very powerful. You basically give a DSL to process json. I can do map, filter and even math.

The reason it's popular partly because it's available from both Centos/Debian based. Just use `yum` and `apt`. Most of time, you maybe limited with what have on a container. And `jq` may even pre-install by DevOps.

Second is it play well with unix tool, pipe and friend.

Yeah, I find jq similar to writing regexes: I always have to look up the syntax, only get it working after some confusion why my patterns aren't matching, then forget it all in a few days so have to relearn it again later.

Any more alternates to Python or Node that are lightweight to add to a project? I'd rather use Node but that's a massive dependency for a non-Node project.

You can use any language with json support, which is most languages.

In python, you can have a jq like experience with

  def f(data):
    # put code here

  data = json.load(sys.stdin)
  output = f(data)
  print(json.dumps(output))

Agree and I also struggle with the docs. The examples are so trivial that they’re useless if you’re trying to use the examples to learn how to use them in context.

I think the examples for any() are any(true, false) and any(true, true). Fair play but not super helpful if I’m new and trying to understand how this fits into a typical jq filter.

I agree, jq is very advanced when everything most people want is simple json transformation and highlighting.

I've made jql[0] for that reason.

Check it out, it's less featureful, but much simpler with a uniform lispy syntax.

[0]: https://github.com/cube2222/jql

Is there anything in my tutorial that I can help clarify for you?

This resonates with me. I don't need more example of basic selects.. json is a tree structure and jq is a tool that allows us to select elements from a tree and transform them.

I would like to see a document that talks about jq from this more abstract perspective and what capabilities exist in this context. For example, can I select two arbitrary branches? Can I merge / move branches? Once we have a grasp of tree manipulation capabilities that are made available through the DSL we can lookup the syntax. Can I select a node and apply filters to it's children? There are so many ways in which a tree can be manipulated and I personally need a higher level description of what I can accomplish with jq.

> [...] but at least I don't need to learn a new programming language (if we can call jq that...)

jq has looping, branching, recursion (with tail recursion optimization), variable assignment, modules, I/O... Yeah, it's a programming language.

I can totally relate.

I think the only way would be a long web page with a massive list of examples to "solve problem X".

But the DSL is ...byzantine, to say the least.

AFAICT, it's basically xpath applied to json. Which isn't helpful, but perhaps explains why it's so obtuse.

And I still have no idea why jq is slow af.

Seems pretty fast to me considering it is reading JSON. JSON is difficult to read quickly.

I know xpath pretty well, and cannot wrap my head around jq's DSL. It might have borrowed from elements of xpath's more obtuse expressions, but simple xpath seems much simpler (to me) than simple jq.

The man page might not be the best way to learn jq. Did you read the documentation on has website? It's pretty comprehensive, with straightforward examples.

Spending one hour or so actually learning jq is a good time investment.

Reply on news.ycombinator.com