jq - like sed for JSON data | Hacker News Reader

jq - like sed for JSON data

stedolan.github.io

132 pointsgnosis29 comments

Considering the amount of json around, wouldn't it be time to settle on a standard for :

- schema definition ( using something like typescript ? )

- data path ( similar to xpath )

- data transformation ( similar to xslt )

i'm pretty sure many people have done their things but i'm still waiting to see a tool like wsdl - based stub generators that would work on many json api.

ps : i know that things like jsonpath and wsdl2 exist, but i really feel like those technologies haven't caught widespread adoption.

I agree and I believe that the way to accomplish this is to use Protocol Buffer schemas.

Protocol Buffers and JSON have an extremely similar data model. The parts of them that people actually use are effectively identical. The actual differences are minor things like:

- JSON can have arrays of arrays; with Protocol Buffers you need a message in between.

- Protocol Buffers can represent binary data directly; with JSON you have to base64 encode it into a string.

- Protocol Buffers are more specific about the size of integer types (int64 and int32 are different); this allows for more efficient in-memory representations but doesn't matter so much on-the-wire.

I think the two are a match made in heaven.

Yes, i also thought of protocol buffer right after sending that comment. Although i wonder : is there any way you could automatically generate a full service/model layer for, let's say angularjs or objective-c, based solely on protocol buffer "standard" ?

You're basically describing Piqi: http://piqi.org/

Thanks for the reference, I hadn't seen that. I really believe, though, that the solution to interoperability is to use protocol buffer schemas directly; not to introduce yet another schema language.

if you need all these , you should use XML. Crockford said he is not interested in any JSON standard extension, at all. not even to put comments in it. That's what XML is for.

Dont use JSON if you should use XML.

I humbly disagree.

The reason i'm using JSON is for the fact that's it's fast to parse, compact, and lightweight, compared to XML. Now the reason for needing the xpath/xslt/xsd version of json is not principaly because it's a serialization format, but because it's a serialization format used in almost any modern web API. Experience shows that using JSON for web APIs is a good choice. Saying "you should've used XML" may have been relevant 5 years ago, but i think it's time to assume the fact that JSON is now so much used, that it needs the same kind of tools that xml has had almost from the start.

I massively disagree. IMHO JSON is more readable, just as powerful and more concise.

Nobody mentioned comments. Nothing the GP listed would in any more be an extension to JSON than HTML is to XML.

Crockford's refusal to extend JSON is genius IMO. Totally agree with you: if the task you're trying to accomplish sounds like a fit for XML, then please use XML. JSON is what it is, and not everything needs to be gradually extended into six mutually incompatible versions of increasing bloat.

http://geekandpoke.typepad.com/geekandpoke/2012/03/thank-god...

And don't use XML if you have requirement where you need to query, insert, modify data pretty often.

Something like SQLite work wells for purposes like these.

> schema definition ( using something like typescript ? )

There is a draft RFC called JSON Schema: core definitions and terminology (draft-zyp-json-schema-04).

> data path ( similar to xpath )

RFC 6901. (JavaScript Object Notation (JSON) Pointer)

I've been using jq for a few months now while working with big GeoJSON blobs and it's a big help. The query language takes a bit of doing at first, but it's really pretty great once you learn it.

curl -s http://tile.openstreetmap.us/vectiles-skeletron/12/656/1582.... | jq '.features[].properties.name'

Personally jgrep seems a better choice here

curl foo/bar.json | jgrep -n -s 'property'

Or even better

curl foo/bar.json | jgrep 'status=500' -n -s 'path'

http://jgrep.org/

For readers who are unfamiliar with sed: don't dimiss jq as yet another json pretty printer. From the jq tutorial "Data in jq is represented as streams of JSON values - every jq expression runs for each value in its input stream, and can produce any number of values to its output stream.".

I've played around with jq and it will definitely become a tool in my toolbox. Kudos to the author.

Also available is the brilliant underscore-cli[1], which brings the functional programming of _.js to the command line.

There's an examples page, if you're struggling to see how it might be useful[2], but I find myself using 'extract', 'find', and 'pluck' on a daily basis.

[1]: https://github.com/ddopson/underscore-cli

[2]: https://github.com/ddopson/underscore-cli/blob/master/Exampl...

recs has this functionality, and _way_ more; worth giving a shot: https://github.com/benbernard/RecordStream

I'm a big fan of RecordStream. Everyone should definitely give it a look.

However, is it true to say that recs has this functionality? RecordStream is great with sequences of relatively simple JSON structures, where you want to count, group, collate, etc. the sequence. It looks like jq is better able to handle large, complex arbitrary JSON objects, pulling out deeply nested fields and transforming them. For example, how would you implement jq's "recurse" in RecordStream?

Now what would be really cool is using the two tools together :-)

And for xml I highly recommend xmlstarlet.

query/transform xml documents from the command line.

Previously on HN (141 days ago): https://news.ycombinator.com/item?id=4985250

Clearly, this is new to enough people that it got to the front page. Merely linking to a previous submission adds nothing here; if you thing the discussion there was informative (clearly not the case here), then say so.

Yeah, I think the previous discussion was informative, it has alternatives and links to ever older relevant posts. Let the karma decide what HN likes. Like you said yourself: https://news.ycombinator.com/item?id=5664116

I like the concept, but how does it compare to using node + underscore?

Isn't the whole idea of JSON that it exactly serializes most JavaScript objects, and the "right way" to do stuff with it is to just use it like a normal object in your program?

The spirit of the tool is to make JSON easier to work with in shell flows. There's plenty of times where you'll get a deluge of JSON and need to work with it (for debugging or whatever).

You may not have a program to use the JSON in.

This assumes you're using JSON in Javascript. JSON is a pretty widely spread human-readable data format now outside of javascript, and it's nice to have good tools that aren't tied to a particular language.

Some time ago I made this: https://github.com/fabriceleal/Jsoned using node.js.

It allows you to use Javascript expressions using a map/filter/reduce paradigm to edit the Json stream, like this:

echo "[1,2,3]" | jsoned -m "' _ * 2 '"

which will print "[2,4,6]"

you can even put the javascript code in files.

Check the wiki (incomplete) for documentation: https://github.com/fabriceleal/Jsoned/wiki/Usage

For a more unix-y version of this, check out Jshon.

http://kmkeen.com/jshon/

Completely unrelated, but I love the typeface he's used on his site. In fact the whole site is laid out really nicely (pretty, yet none of the aesthetics gets in the way of usability. a balance that most sites fail to strike).

Anyhow, tangents aside, this looks a really handy tool. Thanks :)

This is super cool, the language seems both powerful and easy enough to learn. Nicely done.

Reply on news.ycombinator.com