First release of jq in 5 years | Hacker News Reader

First release of jq in 5 years

github.com

114 pointsdrewda27 comments

Great to see. jq is nearly as ubiquitous as bash these days.

I’m excited about pick()! Could have used this so many times:

> Adds new builtin pick(stream) to emit a projection of the input object or array. @pkoppstein #2656

    $ jq -n '{"a": 1, "b": {"c": 2, "d": 3}, "e": 4} | pick(.a, .b.c, .x)'
    {
      "a": 1,
      "b": {
        "c": 2
      },
      "x": null
    }

this looks so, so useful

Delighted to see work continue on jq, which I use for altogether way too many things.

I ask this as a fan of both the project and the new pick() feature: are there any other functions in jq whose interpretation of their arguments is "non-regular" in the sense the PR author means [here][1]?

More concretely, the expression `.a, .b.c, .x` means something very different outside a pick call. It represents a uncollected stream of more than one value, rather than a reduction of an object into a subset of itself.

The syntax inside a pick call is (I believe) impossible to implement in jq; it has to be a builtin. There's no way to get "regular jq" to extract the path expressions as strings with which to set an object's keys.

Ultimately I do think like the addition, but I recognize the tradeoff of "pure" consistency for syntactical brevity. I'd be curious to hear from the maintainers A) whether I'm off-base about the pathexp syntax limitation and B) if there's some philosophy or design principle guiding why and when they might decide to add specialized interpretations of pathexps.

[1]: https://github.com/jqlang/jq/pull/2656#issuecomment-16228220...

Hey, pick is implemented in jq here https://github.com/jqlang/jq/blob/master/src/builtin.jq#L258... using various path functions.

When the jq manual say "path expression" like the argument to pick you can think of it as just normal jq expression but instead outputting values when path/1 is used it will instead output the paths to those values. The limitation is that you only can index into the input (actually including the null value) and don't construct new literals etc.

btw an alternative to pick that might be handy at times is the destruction and shorter object construction syntax:

  # use {a:{b:1},c:2} as input
  # . as {a:{$b}} will bind $b to 1
  # {$b} is shorthand for {b:$b}
  # {a} is shorthand for {a:.a}
  $ jq -n '{a:{b:1},c:2} | (. as {a:{$b}} | {$b}), {a}'
  {
    "b": 1
  }
  {
    "a": {
      "b": 1
    }
  }

I guess I have a terrible imagination. Can someone provide a concrete example of when this should be used?

I could see this being useful when you coded a utility, which runs off a huge existing JSON as its configuration. You want to go straight for one or two values, not this huge data structure.

For me, if I am scripting something, do curl, extract value from returned JSON, continue ...

If I receive a JSON list of JSON objects from an API call, but only a few of the objects' fields are relevant to me, then I can do something like:

  curl https://example.com/api/orders | jq '.data[] | pick(.date, .amount)'

When would this be better than

    curl https://example.com/api/orders | jq '.data[] | {date, amount}'

    curl https://example.com/api/orders | jq '[.data[] | {date, amount}]'

if you want it collected back into an array?

In that case it would be very similar but pick is more general then the shorthand object literal syntax, it can be used with nested index lika .a.b.c or any expression like pick(if .a then .b else .c end) but both has it uses i think.

Is it not an indictment of bash that, in order to make shells useful to modern data formats, we have to create tools with their own scripting languages and pipes? Since bash can't operate on structured data like jq does, jq has to have pipes inside jq expressions.

That's how I'd like it to be. These are domain specific languages, so their own scripting is where I'd see them belong. Command line should stay true to its name: place where you enter your commands (and of course, other things that you can't delegate elsewhere, like variables and control structures).

yeah but JSON has become to de-facto universal format. At the very least, it would be cool if you could convert that into a common runtime format and pipe structured data from one command to another so you don't have to keep parsing things at every stage, or abandoning the shell entirely and working only in jq

ksh93 has "compound variables" that let you have arbitrarily deep trees of data. The syntax for that is ugly. It's hard to use. And I've never had luck with not causing ksh93 to crash when trying to use compound variables.

I think it'd be very nice to have a bash-like shell with a) jq-like JSON values in the shell (really, parsed, jv-like[0]), b) syntax for using jq programs in ${...}, something like ${var@jq:.jq.program.here}.

As for jq having pipes, they're a rather different beast, but I take your point.

[0] `jv` is the internal representation of JSON values in jq.

Yeah, but water is also wet.

Shell scripting is beyond salvation. Inconsistent, illogical crap. It's ridiculous that Bash doesn't have first-class JSON support in 2023, but even if it did, that still wouldn't make me want to write Shell.

I could see a shell that doesn't suck for scripting that has first-class JSON support. Something like PowerShell, but not PowerShell, and close enough to bash that one could quickly get used to it.

Let's call it Javascript!

or node. being able to run external programs is the key here.

Sounds like Elk. https://elk.strct.net

If you haven't come across it yet you might like nushell.

> 1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".

> 2. Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.

> 3. Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them.

> 4. Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them.

https://en.wikipedia.org/wiki/Unix_philosophy

(No, it is not an indictment of bash. This is how tools are supposed to work together.)

Then the Unix shells should be super dumb. Except that wouldn't be very useful.

Hey, this is a release candidate. It got made mainly to test the new GitHub Actions release workflow, though it's just as well because we're very close to doing a 1.7. Probably a week away.

Some competition for https://github.com/itchyny/gojq. I had read somewhere that it was faster than jq - no idea if that's still the case.

It wouldn't surprise me if it is. Same for jaq and others.

Great to see the NUL separator work I had a hand in has reached a release. Wish it got to use -0 for easy typing, but with tab completion it shouldn't be too bad.

Note that this is a release candidate so please report any issues!

Reply on news.ycombinator.com