Comment by weinzierl | Hacker News Reader

Comment by weinzierl | Hacker News Reader

Parent Full thread

jq is incredibly powerful and I'm using it more and more. Even better, there is a whole ecosystem of tools that are similar or work in conjunction with jq:

* jq (a great JSON-wrangling tool)

* jc (convert various tools’ output into JSON)

* jo (create JSON objects)

* yq (like jq, but for YAML)

* fq (like jq, but for binary)

* htmlq (like jq, but for HTML)

List shamelessly stolen from Julia Evans[1]. For live links see her page.

Just a few days ago I needed to quickly extract all JWT token expiration dates from a network capture. This is what I came up with:

    fq 'grep("Authorization: Bearer.*" ) | print' server.pcap | grep -o 'ey.*$' | sort | uniq | \
    jq -R '[split(".") | select(length > 0) | .[0],.[1] | gsub("-";"+") | gsub("_";"/") | @base64d | fromjson]' | \
   jq '.[1]' | jq '.exp' | xargs -n1 -I! date '+%Y-%m-%d %H:%M:%S' -d @!

It's not a beauty but I find the fact that you can do it in one line, with proper parsing and no regex trickery, remarkable.

[1] https://jvns.ca/blog/2022/04/12/a-list-of-new-ish--command-l...

Also highly recommended is gron [0], to make json easily searchable

[0] https://github.com/TomNomNom/gron

Most of the time I can get what I need with gron and traditional UNIX tools, without needing to reach for jq, and without having to re-learn its somewhat arcane syntax.

I came here looking for a gron recommendation. I use this very often.

Is your example not easier to write and read as a 10-something line Python script? I never understood the appeal of jq etc because of this very reason.

I would definitely add dasel to that list. It's become my de facto serialized data converter, and regularly use it to convert between csv, toml, yaml, json, and xml using jq-ish syntaxes.

https://github.com/tomwright/dasel

The yq tool also provides 'xq', which works on XML :)

tbh my biggest problem with all these tools is that I really don't want to have to learn one for each of the json-y formats I have to use every day. If jq supported toml and yaml natively I'd be much much much happier to learn its kind of obtuse syntax.

Have you checked out https://www.nushell.sh/? It seems like exactly what you're describing. Although I know of people who are happily using it as their main shell, I only really use it when I need to read and manipulate data in files.

Hi, lover of jq and author of fq here! Just wanted to mention that fq is jq so you can do things like `fq 'grep("Authorization: Bearer.*") | split("\n") | ...' file.pcap`.

Also i'm working and prototyping some kind of http decoding support that will make things like select on headers and automatic/manual decoding body possible.

I need someone to make a `Q` wrapper that amalgamates all of them. And if that's already taken by a common utility, I vote we name it deLancie, instead.

Reply on news.ycombinator.com