Hacker News Reader

Comment by diehunde | Hacker News Reader

Pardon my ignorance, but would I spend time learning something like jq or zq when it only takes me a couple of minutes to develop a script using some high-level language? I've had to process complex JSON files in the past, and a simple Python script gets the job done, and the syntax is much more familiar and easier to memorize. Is there a use case I'm missing?

View on HN

If you're doing a lot of JSON munging every day and have good mastery of something like jq or zq, you can probably get things done faster.

Like you, I almost always just write Python scripts for such tasks because it's a lot easier for me to reason through it and debug it, but it's definitely slower-going than what I might do if I were very adept in a terse language like jq. I don't do this too often, so it makes little difference to me, but if someone is doing this multiple times a day, every day, it'll add up. As you say, it takes a few minutes; with jq, it could be a few seconds.

The same thing could be said for grep, or really any other utility that can have its functionality reproduced in a programing language.

Indeed! Jq is basically something like grep for JSON.

It might actually make sense to embed jq functionality into your favourite language (as a library or so), as it is quite a nice and well-chosen set of functionality.

I would love to see jq libraries become as common as regex libraries so I could use jq directly in whatever stack or environment I'm working on.

Honestly, I've only really used `jq` to quickly parse JSON structures in interactive sessions e.g

  curl -s http://foo.bar | jq .some.nested.value

Anything more complicated I would indeed go for writing a proper script.

Don't tell anyone, but jq is secretly a pretty well thought out functional programming language.

There is certainly a learning curve with jq that can put people off. The attraction is that the end result is a very small amount of code that does only one thing: parse a JSON file, rather than invoking an external script that might send many HTTP requests or launch a missile.

As the complexity of the input JSON grows or the complexity of your processing, it does makes sense to leave jq behind for a higher level language.

I agree with most of what you say.

I disagree with 'leaving for a higher level language'. Jq is an extremely high level language.

What it is _not_ is a general purpose language.

This is how I felt about regular expression when I was first learning them. Now I feel that they're one of the most powerful text-processing tools that I know. I also felt similarly about SQL at the very beginning. IMO if you find yourself doing a _lot_ of JSON processing, learning at least basic jq gives you superpowers.

1. The High Level Language of your choice may not be the flavour liked by other members of your team. Ruby? ew please use Python - unnecessary discussion ensues... 2. Your High Level Language of choice would probably require a non-trivial container image, which requires extra decisions to be made about sourcing, which is something you'd rather not think about if this is just e.g. a step in a CD pipeline. jq is tiny and a very simple addition to an existing image. It's even present by default in GitHub Actions' `ubuntu-latest`. 3. Your High Level Language of choice may require dependencies to do the same job. How are those dependencies going to be defined, pinned, who's going to be responsible for bumping them...?

I used to 100% agree with you, but these days I understand why so much stuff ends up being bash and jq.

You can spend a few days getting to know jq or you can happily live with your 100+ purpose-built scripts. I know which one I prefer.

I don't even process complex JSON...it's usually pretty basic. But being able to quickly select parts out of streams of JSON data on the CLI is incredibly useful to me, and learning even just the basics of jq has paid for itself a hundred times over by now.

Granted, a lot of my job right now is data forensics stuff, so I breath this kind of stuff. You might never need jq.

I am also on the same side of the discussion, but I'm a programmer by trade. Most of the cases I've seen non trivial jq uses is by people doing command line or shell script magic. In this context I guess it's easier to write the jq expression language than to whip up a fully fledged Python/Ruby/Perl script without having to debug pretty basic stuff once you know the syntax. Pretty much like awk.

I'm a programmer by trade. I use jq. :)

jq is great for shell scripts. Say your script hits an API that returns JSON, and you want to retrieve a single field. This can be difficult to do correctly with grep or other text matching tools, but is trivial with jq. You just pipe it in like "curl XYZ | jq '.path.to.your.data'"

I imagine this is how it's used 90% of the time, but can do lots more advanced stuff as described in the article.

Suppose you write a shell script which is intended for use among colleagues as part of a pipeline.

In many cases, the most appropriate and useful tool for the job would be jq - one line in the shell script corresponding to the required data transform, calling out to `jq`, which already has a reasonable user base and documentation, and could be trivially replaced by anyone if the business needs change.

Reply on news.ycombinator.com