Curl will now output JSON | Hacker News Reader

Curl will now output JSON

daniel.haxx.se

120 pointslogicalstack28 comments

I think if all command line tools outputted JSON, it would go a long way towards fixing the "ad-hoc parser problem" in scripts, and bring unixy shells around to a more PowerShell-like object-passing style. Passing plain text around is a great idea, but It's silly to restrict yourself to a single structural unit within that text (the "line"). We have better tech now.

While powershell is a lot more then passing hashtable around, it would make a first good step. Accepting json object as parameter in "unix standard" should be done along.

Yep this would be really great.

Have you seen jc[0] as an effort to wrap common utilities to output json?

[0]: https://github.com/kellyjonbrazil/jc

I hadn’t; cheers!

It's nice but I don't think json is such a great format for it. The overhead of parsing it is incredibly annoying along with being unable to pack binary data which is what unix tools can deal with to some extent

json is fast, according to https://github.com/xi-editor/xi-editor/blob/master/README.md... "I considered binary formats, but the actual improvement in performance would be completely in the noise. Using JSON considerably lowers friction for developing plug-ins, as it’s available out of the box for most modern languages, and there are plenty of the libraries available for the other ones."

It doesn't have to be a one size fits all solution, there are edge cases where json wouldn't be the right choice, but for many and most things i could think of that i use daily I'd be happy with json. If it's small binary data i guess it could be base64 encoded.

Querying network interfaces, directories, process information, systemd daemons or anything else, i see these things could do great if they just outputted json and my shell could treat it as an object. If a script is performance critical, maybe it's time to rewrite it in another language anyways?

i have 2 letters for you: jq

any tool that outputs json + jq => instant scriptability win.

Particularly if you prepend queries with try. jq likes to throw errors when unexpected things happen in your search or your data, and it may return "null". Using try instead returns an empty string.

When walking JSON structures to validate data I code each validation separately instead of burying them in blocks of if-then-else logic. I think this makes it easier to maintain and more obvious when dealing with data that for example may be in a list or an array of lists.

But yeah if you write golang scripts and want to shuffle data in and out of bash, jq is your #1 power tool.

Two letters, but then several lines to write an actual simple request (with some dances to battle the null and the exit code), and it's slow.

libxo is doing a good job for FreeBSD. It would be nice if Linux also adapts that, or something similar, one day.

Eg I have a script that needs output from netstat, and `netstat -bin --libxo json | jq ...` is a great way to get that info instead of scraping the human-readable output.

Sorry, stupid question, but why not use /proc/net directly?

/proc is a linuxism.

Yes... But even on Linux I've been bitten so much by changes in 'standard' tools just on successive versions of centos, the last 10 years... ifconfig changing its output syntax (I know 'just use ip' well it changes sometimes too...), hostname sets a transient hostname now, hostnames with _ are now refused... netstat had some subtle changes some time ago too. free changed too. df, fricking df output format changed. So now, I just hit /proc and /sys, and document the hell out of it. Sure, new fields can appear there but they're new columns. The parser has to handle 'additional' columns. I trust Linus more with not-breaking-userland than the folks that make 'portable' tools.

Glad I don't have to support other OSes...

It would be great to have libxo in linux!

FYI, `jc netstat` also generates JSON output for `netstat`.

(disclosure: I'm the author of `jc`)

While I'm at it... check out `jtbl` on my github which prints the JSON or JSON Lines output to a table in your terminal.

https://github.com/kellyjonbrazil/jc

https://github.com/kellyjonbrazil/jtbl

Maybe a good interim solution would be a “proxy command” that knows the output format of common commands, and how to convert them to json. So for example you could pipe the output of ls to this tool, and it would be smart enough to parse it and convert to json.

https://github.com/kellyjonbrazil/jc was posted in another comment; it appears to be exactly that.

It's slow and obtuse, the syntax of processing commands is too verbose, there are surprises with null values and return codes jq gives.

It's not the answer for everything, it's just one of the tools. It feels cosy to huge amount of people because it's the product of web explosion, but at the core it's accidental and not that good.

No need to panic! You can still pipe it to gron [0] and recover useable output.

[0] https://github.com/TomNomNom/gron

Note that this is not titled, "Curl will now format JSON". This will "Make curl display information on stdout after a completed transfer"[1].

[1] https://curl.haxx.se/docs/manpage.html#-w

I'm testing it, maybe I'm doing it wrong but I don't understand well if it's possible to show only the json output (and not both json and response body). I'm doing it with a trick:

curl -I -XGET -w '%{json}' https://www.google.com | tail -1 | jq

    -o /dev/null

It is a step in the right direction, but still far from all the goodnes of PowerShell's Invoke-WebRequest . Not sure curl can do anything about it though.

Do you still need to pass it to `jq` for it to be pretty printed or no, I couldn't tell from reading the article?