Comment by nonlogical | Hacker News Reader

Comment by nonlogical | Hacker News Reader

JQ is an insanely powerful language, just to put to rest any of your doubts about what it is capable of here is an implementation of JQ... in JQ itself:

https://github.com/wader/jqjq

It really is a super cool little, super expressive nearly (if not entirely) turing complete pure functional programming language.

You can:

* Define your own functions and libraries of functions

* Do light statistics

* Drastically reshape JSON data

* Create data indexes as part of you JQ scripts and summarize things

* Take JSON data, mangle it into TSV and pipe into SQLite

  cat data.json | jq '<expr>[]|@tsv' | sqlite3 -cmd ".mode tabs" -cmd ".import /dev/stdin YourTable"

And also for prototyping you can also use it to tailor output of APIs to what you need in a pinch, using JQ as a library especially with something like python:

https://pypi.org/project/jq/

As a part of the library you can compile your expressions down to "byte-code" once and reuse them.

Saying JQ is a best kept secret is an understatement. JQ gets more amazing the deeper you dig into it. Also it is kind of crazy fast for what it is.

edit: Formatting fixes

JQ + journald is great too, but 20 years of muscle memory writing bash / python / perl / awk / sql / ruby / JS / CSS selectors / xpath / xmlstarlet one-liners keep getting in my way. I keep long notes on both with examples of common tasks. I still dislike yaml (significant whitespace is my “ick” as the kids say) too much to learn whatever the equivalent is for that and still find CSV/TSV easier to slice and dice at will due to my own personal history.

I’m sure at this point that many ETL jobs in notebooks we run at $BigCo today could be reduced to jq expressions that run 100x faster and use 1/10th the memory.

The ‘nearly’ Turing complete is something I wonder about. It feels like jq might have some limitations - transformations it can’t do, due to some inherent limitation of how it handles scope or data flow. The esoteric syntax makes it hard to determine sometimes whether what you are attempting is actually possible.

As soon as jq scripts reach a certain level of complexity I break out to writing a node script instead.

And given how rapidly jq scripts acquire complexity, that level is pretty low. One nested lookup, and I’m out.

jq does often feels like a code golf language. I would say it does have some of those Perl one liner vibes, that is to say that it is often a write-only language.

Also the ‘nearly’ part is because I don’t remember if it has infinite loops or if it is more like Starlark and thus decidable. I do have vague recollections of causing infinite cycles in JQ, it quite as well could be entirely Turing complete.

So far I have not found a single task that JQ was incapable of. And I have abused it pretty bad on my spare time =], for intellectual challenge.

jq lacks coroutines, which means some tasks can be hard to accomplish in jq. It's still a very powerful language, and it is Turing complete, not just nearly.

Thanks for the jqjq shoutout! :) i'm quite sure jq is turing complete, jq (and jqjq!) can implement brainfuck https://github.com/01mf02/jaq/blob/main/examples/bf.jq

Thank you so much for piecing together a great example (jqjq) to help open everyone’s eyes that JQ is not just a JSONpath implementation with weird syntax! I often reference it to drive home the fact that JQ is a full blown language.

The brainfuck one is also gonna be going into my notes. That implementation is quite a terse implementation.

Great to hear and that was one of my hopes! but honestly it initially came to be because i was fiddling with some jq AST-tree stuff for fq :) weirdly it was much easier to implement than i expected. Hardest part was how to handle infix operators, +/- etc, parsing without infinite recision. But once i found and managed to implement precedence climbing things got a lot easier, it's still a bit of magic to me how well it works :) the eval part had some difficulties but mostly straight forward when you can piggy-back on the "host jq", but i tried to stay away from piggy-back too much, to not piggy-back at all probably requires implement a VM somehow.

BTW your very welcome to help improve the jq documentation. Me and some other maintainers have been talking about that it probably needs an overhaul to be more approachable and also better document some nice hidden features. Join the discord if you want!

Reply on news.ycombinator.com