Jqjq: Jq Implementation of Jq | Hacker News Reader

Jqjq: Jq Implementation of Jq

github.com

248 pointspetercooper63 comments

Open articleView on HN

This is up there with the scariest piece of software in existence:

https://github.com/xoreaxeaxeax/movfuscator

Presentation of idea:

https://m.youtube.com/watch?v=R7EEoWg6Ekk

Based on mov is Turing-complete by Stephen Dolan, also the author of jq. Small world.

Is this just a coincidence or was the grandparent aware of this connection?

Bootstrapping in itself is already a somewhat magical concept even when done in a serious way. I remember the first time I wrote a compiler quite distinctly. There is something special about building something that can build itself. I think it embodies the potential of programming in an absolutely unique way.

TIL GitHub highlights **Warning** in blockquote https://github.com/community/community/discussions/16925

At first I asked "why??" and then I remembered what site we're on. I need to start having fun with tech again.

Aye, computation is a medium of expression.

You mean computation is a medium of computation

Expressions are a medium of computation.

Jq is ok, but the language is awfully unintuitive. It's also not very user-friendly. I've been using gojq with better success

Yeah, I like the design goals, I just found the solution was worse than the problem it was solving.

I'll never be intelligent enough to use jq properly.

I just use gron then grep for what I'm looking for.

Does it do more than just grep? That's awfully limited compared to what jq can do. I personally like jless for just searching and exploring.

gron is great, i have this in my init.jq file for fq (https://github.com/wader/fq). Sadly jq lacks a path_to_expr function so a bit harder to do it with that.

  def gron:
    ( path(..) as $p
    | getpath($p)
    | scalars
    | "\($p | path_to_expr) = \(tojson)"
    | println
    );

Ex:

  $ fq gron <<< '{"a": [1,2,{"b": 3}]}'
  .a[0] = 1
  .a[1] = 2
  .a[2].b = 3

Great for copy/paste into other expressions when poking around.

I’ve been using ‚gron’ with good results

How is gojq different? Cursory look at the README and it seems like it implements the same language.

My problem is the infrequency with which I need to use jq means I don't create a good learning feedback loop. The last time I used it though, I was able to walk down a decently complicated structure (indexed val in an array in a key in a hash in an array) and extract the data I wanted only having to google once. Like awk, though, I know there's a world of functionality I'm just not using.

You should checkout gron, it is great at flattening fragments of json.

https://github.com/tomnomnom/gron

I have started using gron to understand the json structure to then write jq. Other times it does the job alone. It’s great.

You can pipe the output of gron into fzf, it makes it kinda nice for perusing the flattened json.

TIL. Thank you! I took a look and will definitely incorporate this in my work. Definitely a good idea.

Anytime jq is mentioned, gron should be mentioned shortly thereafter. It is a wonderful tool.

It is indeed a reimplementation of jq in go, and the reason I use it over jq is many, one of which is that it produces more helpful error messages. In fact, I've moved jq to ojq (sometimes I want to check compatibility), and ln -fs gojq jq.

I feel like it's OK if you use it often, but when you need it once a year, I'm spending too much time trying to figure it out.

Some collection of typical problems and solutions would help I guess.

I use it a bunch to string together web APIs, and you're absolutely right. The longer I go relying on my library of pre-written strings of commands, the harder it is to write a new one. And you're right again with your second point, because that same library makes for great reference material for myself.

The manual (https://stedolan.github.io/jq/manual/) somewhat provides that. Quite a few sections have helpful examples, with links to the sandbox at jqplay.org.

The IMO not so good parts of that page are

- Quite a few section headers don’t make sense to novices

- I think the table of contents would be better if it had subheaders, ideally with one-line descriptions of the features (that probably would solve the issue above)

- not every section has examples

- some examples use fairly advanced features, making them harder to understand then necessary

I see folks have mentioned gron, and I also want to suggest:

- jello (PyPI) - yamlpath (PyPI) - dasel

jq is not OK for that exact reason. It's like sendmail, dividing the world into people smart yet obtuse enough to sit and learn its idiosyncrasies, and regular people that go mad and claw their eyes out whenever they try.

I tried, more than once, but I guess I'm not smart enough.

Have jq always been a fully-capable turing-complete language? Or did it become this monstrosity step by step, somewhat naturally? I wasn't even aware until today that it's… uh… a programming language.

I think it has been turing complete for a long while. Here is the first prototype in haskell before it was convert to C https://github.com/stedolan/jq/commit/eca89acee00faf6e9ef55d..., it's very basic but i would guess it has enough things to be turing complete.

... Can I use it to run jqjq?

Yes you can! but it's very slow

  $ ./jqjq "eval($(jq -Rs . jqjq.jq)+.)" <<< '"eval(\"def f: 
  1,8; [f,f] | map(.+105) | implode\")"'
  "jqjq"

But running jqjq in jqjq in jqjq runs out of memory so don't know if that works yet :)

very impressive. I like how github even says language: jq 100%.

This is genius and horrifying at the same time, I love it!

How does this language implementations in the language itself works? Isn’t it trivial to implement any feature, given it’s already implemented? They use lower level features to implement higher level ones?

Sure, you can implement the primitives by just passing through to the host runtime if you want, but you still have to parse the source and walk the AST first.

Impressive. Someone should do jq-jit too. I briefly looked into it and it seems like it would be _a lot_ of effort :/

I'm not sure why jq has its own language which is not js based.

At least xmlstarlet has xslt for language.

Hi author of jqjq here. I don't know why Stephen Dolan designed the jq language the way he did. But for me the most attractive things about jq is the terse and pipe-friendly syntax. This allows you to write queries in shells and REPLs without the need of multiple lines, and it's easy to compose add addition pipes. And by using generators/backtracking you can do complicated traversals over tree and list without much syntax.

Now after using jq a lot think about it the other way around. The language itself is no that tied to JSON, it can be used for more things with other types then just JSON. If you interested have a look at https://github.com/wader/fq where i've tried to expand on it.

I have been using gojq, so all I have to do is scp the gojq binary.

SCP doesn't work?

Has this happened to you? You're logged into a remote server and trying to read a config file or log file. The problem is its in a minified JSON format and looks like a solid wall of noise when you try to use `less`. You reach for your go-to JSON swiss army knife `jq` but it's not installed! You don't have root access to install it via `apt-get` so you need a script you can run locally. Just copy the `jqjq` and `jqjq.jq` files to your local directory, and now you can access the full power of jq just by running `./jqjq`. Problem solved! And the best part is that the only dependency is `jq`, which should be installed pretty much everywhere.

> And the best part is that the only dependency is `jq`, which should be installed pretty much everywhere.

I am confused by this sentence. Wasn't the entire premise that js is NOT installed?

I believe that that was the joke

You understood the joke, you just didn't find it funny.

Jokes aside, this is a actually a pretty common scenario for some people, and I wrote a tool called Exodus to solve it [1]. Exodus makes it trivial to relocate Linux binaries like jq from your local machine [1]. You can simply run

  exodus jq | ssh my-server

locally, and then immediately have access to jq on the server. I personally find it very useful when working with servers or containers that have limited package repositories or where I don't have root access.

[1] - https://github.com/intoli/exodus

Ooh. Thank you for Exodus! It is fantastic!

>Exodus handles bundling all of the binary's dependencies, compiling a statically linked wrapper for the executable that invokes the relocated linker directly, and installing the bundle in ~/.exodus/ on the remote machine.

That's really neat.

That is very cool, and it's going to help me alot on old RHELs I think. Looks like it was actually very complicated to make!

Discussed 11 months ago:

Exodus – relocation of Linux binaries–and all of their deps–without containers

https://news.ycombinator.com/item?id=29446297

(307 points, 65 comments)

In all seriousness Python's json.tool is a great option here.

    cat big.json | python3 -m json.tool

    python3 -m json.tool big.json

There's also json_pp(1), provided by Perl.

Joking aside

    ssh badly-managed-server cat file | jq .

Problem solved !

I actually made a wrapper that just guesses (based on which parser returns error basically) type of serializing and then displays it nicely.

Thanks for this. The pattern is useful for much more and sure beats pushing ad-hoc 3rd-party-relinked binaries to production containers.

Then again, why not scp the file to local? Bandwidth usage isn’t good for the environment.

Why stop at one file, is it possible to locally mount a remote folder over ssh?

Yep, it's called sshfs, very easy to use.

here is function I use

    sshmount () {
     SSHFS_SERVER_DIR='/home/user' 
     if [ "z$1" = "z" ]
     then
      echo "specify server [server_dir]"
      return
     fi
     SSHFS_SERVER=$1 
     if [ "z$2" != "z" ]
     then
      SSHFS_SERVER_DIR=$2 
     fi
     SSHFS_TARGET_DIR="/home/user/mnt/$SSHFS_SERVER" 
     echo "Mounting $SSHFS_SERVER:$SSHFS_SERVER_DIR into $SSHFS_TARGET_DIR"
     mkdir -p $SSHFS_TARGET_DIR
     sshfs -o idmap=user -o allow_other -o kernel_cache $SSHFS_SERVER:$SSHFS_SERVER_DIR $SSHFS_TARGET_DIR && cd $SSHFS_TARGET_DIR
    }
    
    sshmount servername

will mount server's home in directory /home/user/mnt/servername, second parameter changes the server dir to specified one

You can use cut, tr and sed

https://stackoverflow.com/a/70614580/1507124

it's not full pretty, rather just newline delimits arrays. Incidentally, it's much faster than jq. Probably wouldn't use it for anything serious

Just out of curiosity, why would anyone want to store a config file in a minified JSON format?

Probably computer generated and defaulted to compact output.

Bravo; I got most of the way through before catching the joke:)

> Your scientists were so preoccupied with whether they could, they didn't stop to think if they should

Is it slower than the reference Jq written in C?

Reply on news.ycombinator.com