Q: A faster re-implementaiton of jq written in Reason Native/OCaml

https://docs.microsoft.com/en-us/powershell/module/microsoft...

It is not nearly as expressive as jq, but it is faster for my use cases (written in golang).

https://github.com/borkdude/babashka

https://news.ycombinator.com/item?id=24353476

Aside: another nice tool I recently discovered for working with JSON and YML, doing conversion and diffs (especially helpful for generated files):

https://github.com/homeport/dyff

> Make JSON greppable!

> gron[1] transforms JSON into discrete assignments to make it easier to grep for what you want and see the absolute 'path' to it. It eases the exploration of APIs that return large blobs of JSON but have terrible documentation.

  ▶ gron "https://api.github.com/repos/tomnomnom/gron/commits?per_page=1" | fgrep "commit.author"
  json[0].commit.author = {};
  json[0].commit.author.date = "2016-07-02T10:51:21Z";
  json[0].commit.author.email = "mail@tomnomnom.com";
  json[0].commit.author.name = "Tom Hudson";

    echo '{"foo": "bar"}' | query-json ".foo" /dev/stdin

He replied and thought about qj

http://harelba.github.io/q/

Query CSV with SQL

[0]: https://github.com/cube2222/jql

It is more verbose, like you get the size of something with array:size or map:size functions, so it is more readable

I am implementing it in Xidel 0.9.9+: http://www.videlibri.de/xidel.html

I have a issue to improve performance where I can push this forward: https://github.com/davesnx/query-json/issues/7

But sure, are caveats!

https://github.com/davesnx/query-json#performance https://github.com/davesnx/query-json/blob/master/benchmarks...

But all explanations aren't based by any evidence, just asumptions.

IMO, an individual dev making a fast useful tool should always be welcomed as a feat of worthy hacking.

But if I had done something like that, and then serendipitously discovered that I was exceeding the original's performance, I certainly wouldn't be shy about it.

Also, this comes across as armchair criticism purely for the sake of armchair criticism. My own experience has been that, when I'm doing ETL that involves wrangling JSON, the "wrangling JSON" bit of it is almost always the bottleneck. So any improvement is more than welcome and deserves to be cheered. Even if it's an improvement on something that's already the current fastest way to do it.

Reimplementing a piece of software that is 12 years old which mimics their UX and improves performance and error messages it's more than welcome in my opinion. My purpose was to learn the OCaml stack of writting compilers, so I personally found that I "needed" a language already created.

Thanks for raising those concerns

We'd all be better off if plebes grew their skills by reimplementing common tools.

I used Reason to compile to Native, so using OCaml's stdlib and OCaml's dependencies and compiling it with OCaml, but my source code is written in Reason syntax.

In this case the author is using Reason as an alternative syntax to OCaml. Reason resembles javascript a little more, and some people find that nicer to work with. So the idea is that you write Reason code, then translate it into OCaml code using the Reason tools, and then ultimately you compile it down to a native binary.

If instead you want to write a web-app which runs in a web browser or node.js, then you'd need to compile it to Javascript, which is what bucklescript helps you do.

Where does Rescript come in? As explained above, Reason can be used for writing either native apps or javascript apps. However, it's hard to evolve the syntax of Reason in a way which satisfies both aims. So they've now split the work -- going forward, Reason will specialize on native, and Rescript will specialize on javascript apps. Their syntax is expected to diverge from each other, in order to support those aims as best as they can.

It wasn't.

Before doing any curl | bash, check what's on the install command, that's the entire point of it.

I was somewhat surprised it didn't use an existing json parser library.

I often have to pluck out attributes from streams of json records (1 json object per line) - often millions/billions.

jq is almost always the bottleneck in the pipeline at 100% CPU - so much so that we often add an fgrep to the left side of the pipeline to minimize the input to jq as much as possible.

The main weakness seems to be streaming use cases (not having the whole file in memory at once). These are supported, but the syntax is quite awkward.

The C code is clean enough as C code goes, but fairly monolithic. And it’s C, so it’s not noticeably slow until you start processing GB. But it would probably take a rewrite to improve its performance significantly.

and there are a few techniques to "discover" the schema of the json file, I trend to read with '.' or 'keys' and later keep going.

I'm planning to implement a flag where each operation prints the internal state of the json, so you would see what are the "pipes".

I will pick a few of your cheatsheet to implement next in q, Thanks!

Menhir/sedlex and others are pretty high accessibility barrier for new commers.

One of the nice things about all of it it's the discord, it's friendly and always helpful.

Hope it helps, just let me know if there's any specific!

For the set of operations that I implement it it's faster, that's true.

Replacing that with, say, traditional command line flags would make it a lot less useful for me, I'd probably have to build much longer pipe-chains to do things that are relatively simple and readable jq snippets (if one knows the syntax.)

Using an established scripting language in its place would make it pretty much just python -c/ruby -e or whatever with some pre-loaded functions, but what's the point? You can always just write a quick python/ruby/whatever script, jq to me is an alternative for cases where a script feels unnecessary. It would also mean everything gets more verbose, so less of my jq transformations can be inlined without loss of readability.

Aligning it to more established languages would probably cause confusion as well in those cases where it doesn't match the reference language 1:1. Looks like javascript, writes like javascript, but only for a tiny subset of the language, etc.

Doing this only for a few function names or syntax constructs still results in a pretty unique and unusual language that will require people to reference the docs a lot, just now lots of existing scripts break.

The feature that I think penalizes a lot jq is "def functions", the capacity of define any function that can be available during run-time.

This creates a few layers, one of the difference is the interpreter and the linker, the responsible for getting all the builtin functions and compile them have them ready to use at runtime.

The other pain point is the architecture of the operations on top of jq, since it's a stack based. In query-json it's a piped recursive operations.

Aside from the code, the OCaml stack, menhir has been proved to be really fast when creating those kind of compilers.

I will dig more into performance and try to profile both tools in order to improve mine.

Thanks