Tips on adding JSON output to your command line utility. (2021)

I would argue it's not just about familiarity. I had the same reaction to that quote and feel the same as you do, and yet I have been using jq before I first learned and started using awk.

For a long time I had been familiar with jq, but every time I had to use it for any non-trivial task after not touching it for a while I had to go to the docs and really scratch my head. Meanwhile, I had no idea about awk, and for ages I would just see those awk one-liners on Stack Overflow as weird arcane alternatives to "simply" piping sed, grep and tr over bash.

However, one day I finally had the time and the reason to have a look at AWK Programming for a specific use that I needed, and it immediately clicked. Awk makes sense and now I can easily fall back to it every time, even if I haven't used it for ages. At most, all I need to look up is the order of parameters in the gawk regex functions.

And yet, time and time again I am baffled by jq, however much I have used it in the past. Whenever I need to parse a JSON I'm happy enough if I can use jq to get it to a good enough halfway point so that I can pipe it to awk to do the actual work.

ChatGPT flawlessly constructs jq commands for whatever banal task you have. Game changer.

I output JSON and embed a jq-like arg in in all of my projects (https://rotx.dev/docs/references/cli#jq). Makes it really easy to slice and dice output.

It's not you; parsing line-based text is fundamentally simpler than parsing structured data like JSON, especially nested JSON. It's just a lot easier to reason about, and a lot easier to split up in smaller chunks, too. Who hasn't ended up with something like:

  cmd | sed | grep | sed | grep

Each bit gets you closer what you want in small easy-to-understand steps, making it fairly easy to develop. And for things you only do once or twice, who cares as long as it works, right? I find that's a lot harder to do with jq. I feel this is a problem waiting for a better solution (and gron isn't it).

I've been using sed, awk, and friends for I find myself reaching for 30+ years, refer to jq documentation 90% of the time I use it (vs. <<10% when using traditional UNIX tools), and still appreciate JSON output as an option.

Incidentally, PowerShell has nice built-in facilities for working with JSON:

  /Users/jtm> (
  >>   diskutil list -plist `                                                                        
  >>     | plutil -convert json -o - - `
  >>     | ConvertFrom-Json
  >> ).AllDisksAndPartitions.APFSVolumes `
  >>   | where MountPoint `
  >>   | select `
  >>       VolumeName,`
  >>       @{L='Size';E={$_.Size/1GB}},`
  >>       @{L='Free';E={($_.Size - $_.CapacityInUse)/1GB}} `
  >>   | where {
  >>       $_.Free / $_.Size -lt 0.10
  >>   } `
  >>   | sort VolumeName `
  >>   | ConvertTo-Json
  [
    {
      "VolumeName": "Gríðarvölr",
      "Size": 1862.8213348388672,
      "Free": 150.1973648071289
    },
    {
      "VolumeName": "Mjölnir",
      "Size": 1862.6963348388672,
      "Free": 92.34595108032227
    }
  ]

I've definitely waxed & waned a number of times through jq familiarity. And sometimes it is a bit of a stress.

But this feels like a vastly unfair basis for comparison. None of the old tools do 1/1000th what jq does. Arbitrary structure data in, a whole pipe of simple or complex commands, structured data out. Jq is complex maybe sure yeah, but it replaces some really gnarly inefficient awful zsh scripts I'd have to cobble together with something that, in the end, is sensible & clear & direct. That other people would probably also write the same way I did.

Living in the structures data world is so much better than arbitrary text-line oriented systems. They may individually be better, but the utter lack of consistency in the old world meant reinventing too much, each time you want to do a task. Jq effectively is a mini shell, full of shell like concepts (operators, variables, pipes,...), and on those grounds, it's not like zsh has any shortage of capabilities most people don't know about & use well. Most of our shell scripting is beating on the same 10% and having to go hunt around through the other 90% on demand too.

have you tried gron?

I have tried many alternatives, even including large Go binaries such as gron. None of them appealed to me. I use a comparatively simple C program that is 45.9 KiB when statically-compiled. Of the alternatives I have tried, this one was the most interesting:

https://news.ycombinator.com/item?id=31166956

An array of records, or just 1 record per line, also works nicely with jq.

For that matter, it can even be imported in like duckdb and handled like a table :)

This is really the only reason, because JSON as a format is otherwise terrible. No good way to represent 64 bit numbers, very limited types and no comments.

Since yaml is a superset of json, sometimes when I have to use yaml for some tool/config I just use json (and optionally with yml comments).

This is self-contradictory. JSON is more typed and more predictable than what comes out of UNIX tools.

No, parsing is easy...

I recommend the "Crafting Interpreters" book.

> I worry we're essentially reinventing a less verbose form of SQL though.

We already have that with paste, join, etc. from Coreutils https://learnbyexample.github.io/cli_text_processing_coreuti...

Why is that important?

.

    --json

it's one character shorter than

    -o json

Not really that crazy at all. And yes it's definitely good.

Newline-delimited JSON is so much more useful to me than weird Unix line-formatted output that I have to parse with pattern matching or regular expressions.

It's basically a line-based format that can represent all of the JSON types and includes support for nested data structures where necessary. What's not to like?

If you're using UNIX tools to parse it, it sucks, but generally if I'm reading the output of a command, and the command is more than one word, I'm doing the whole thing in Python.

That's real programming, and for that I want type checkers and debuggers and modern syntax and all that. And the performance is often faster because you're not spinning up subprocesses for each command.

./some_command | jq '.memory_use'

vs:

okay, run it once with head (assuming we have a --dry option...) ah, that's the column. okay cut -d"," -f0, ah whoops it's starting at 1. ah, damn, there's a weird comma in the name/quote. oh weird, that one's null, ah heck.

schemas are cool. JSON extends and builds on the UNIX idea of having things pipe and plumb well together.

Line-oriented formats, like most traditional Unix-style tools, are for human consumption. JSON is bad at that, thus gron.

On the other hand, structured output formats, like JSON, make it easier to consume with other programs. Standard formats have readily-available and commonly used libraries, whereas line parsing tends to be one-off for every program. Whether JSON is the best format for this is certainly debatable, but it is quite ubiquitous, which is a huge advantage. I doubt many folks would propose XML as a general recommendation.

Tools should have both options on their path to maturity -- both human-consumable and computer-consumable output format options.

I’d rather use JMES Path (or jq even tho it’s messier) to restructure my data rather than some mixture of awk, sed, cut, etc.

Line output often needs restructuring in my experience, though JSON will always need it.

Unix tools on newline delineated list like objects work great, but try them on dict like objects and it becomes clunky really fast. Parsing `ip` output is a good example where the data naturally lends itself to iface:attrs. Plucking the value you want by `.[.name | startswith('eth')].addr` is way easier than pulling this out with grep/awk.

The caller can just specify the output format that they want, and ever since I switched to nushell I pretty much always want json.

What orthogonality? “Line-based data” that varies randomly from tool to tool? Using json between programs is perfectly orthogonal.

Nonsense. Using a structured format means all tools are using the same format, instead of every tool making up their own (usually broken) system.

Look at how many flags `ls` has to feed filenames into other programs without breaking.

Also, using a proper format like JSON makes it actually robust. Most ad hoc pipelines break if you so much as put a space in a filename.