wwader

I see, thanks for replying and no worries! yeap some of the "self-describing" formats like msgpack, cbor etc will because of how fq works have to be decoded into something more of a meta-msgpack etc.

About blobs, if you want to change how (possibly large) binaries are represented as JSON you can use the bits_format options, see https://github.com/wader/fq/blob/master/doc/usage.md#options, so fq -o bits_format=md5 torepr ...

I can highly recommend to learn jq, it's what makes fq really useful, and as a bonus you will learn jq in general! :)

hope it can be of use!

Hey, fq author here. Happy to hear it's useful! could you elaborate a bit more how it was less intuitive? fq's query language is jq with some small additions so i wonder if you might mean the decoded structure is more detailed/verbose as it includes all the "low level" details? maybe your looking for the "torepr" function that converts the detailed structure into the "represented" value?

Ah sorry that is probably zsh specific, it makes sure to not glob expand * etc

I do quite a lot of adhoc/exploratory programming to query and transform data then jq is very convenient as it works very well with "deep" data structures and the language itself it very composable.

To debug in jq you can use the debug function to prints to stderr, ex: "123 | debug | ..." or "{a:123, b:456} | debug({a}) | ... " only prints value of a "{a:123}"

Just had to try:

  $ function _select_aux () { clickhouse local -q "SELECT $* FORMAT Pretty" }
  $ alias SELECT='noglob _select_aux'
  $ SELECT COUNT(*) as count FROM file('repos.json', JSON)
  ┏━━━━━━━┓
  ┃ count ┃
  ┡━━━━━━━┩
  │    30 │
  └───────┘

Interesting post! when prototyping what would end up being fq i did quite a lot of tinkering with how it could work by splitting it up into multiple tools, and use it like: <decode tool> | <query tool> | <display tool>. Never got it to feel very neat and problem is what would be piped? i tried JSON so that <query tool> could jq but whatever is would be would have to be quite verbose to be able for <display tool> to be able to show hexdump dumps, value mappings, descriptions etc. So in the end i ended up doing more or less the same design but in jq where the values piped between filters is kind of a "polymorphic" JSON values. Those behave behave like JSON value but can be queried for which bit range and buffer they originate from or if the value is symbolic representation, description etc. Maybe an interesting note about fq is that for format like msgpack it kinds of can decode in two modes, by default it decodes msgpack into like a "meta"-mode where things like integer encoding, sizes etc can be seen and then there is another "representation"-mode that is JSON value of what it represents.

Hey! fq author here. I have a bunch of related tools in the readme https://github.com/wader/fq?tab=readme-ov-file#tools two suggestions: gnu poke and wireshark (can decode lots of more things then just network protocols)

Yeap i've talked to itchyny quite a lot about various changes https://github.com/itchyny/gojq/issues/153 and also upstreamed quite a lot https://github.com/itchyny/gojq/issues?q=author%3Awader like custom iterators (to allow eval, own iterators and "empty" functions), query marshalling (query rewrite tricks) and a bunch of small things and bug fixes. But the largest change to add a JQValue interface is quite complex, other changes like extended literals is also a bit tricky.

Hmm weird list of changes for https://github.com/wader/gojq/compare/fq...itchyny:gojq:main but i guess it is because i haven't kept my main branch in sync. The fq branch should be based on latest gojq/main as of now. I usually try to rebase as quick as possible.

Let me know if you have any other questions or want to help out! maybe email etc as i usually don't check HN comments replies that often :)

Did someone say let's represent structured data as json? a bit of shameless plug: https://github.com/wader/fq :) It's using a fork of gojq btw!

I think https://pkg.go.dev/encoding/json#Decoder do support steaming at least. Here is gojq's stream mode https://github.com/itchyny/gojq/blob/main/cli/stream.go

Feels fishy and generated somehow. Quite sure there is no @pretty format in jaq or jq and the section about map function does not use map function.

What might be confusing is that iterating an object iterates its values. add is defined something like this: def add: reduce .[] as $n (0; . + $n)

If you spend lots of time with certain binary formats then i can recommend adding a decoder, happy to help with it also!

https://github.com/wader/fq has a REPL and can read JSON. Tip is to use "paste | from_json | repl" in a REPl to paste JSON into a sub-REPL, you can also use `<text here>` with fq which is a raw string literal

Quite a lot! i use it to explode both JSON and tex (parse using jq functions). I also use it for exploring ane debug binary formats (https://github.com/wader/fq). Now a days i also use it for some adhoc programming and a calculator.

jq 1.7 do preserve large integers but will truncate if any operation is done on them. Unfortunetly it currently truncates to a decimal64 which is a bit confusing, this will be fixed in next release where it follow the suggestion from the JSON spec and truncates to binary64 (double) https://github.com/jqlang/jq/pull/2949

Yeap was fixed in 1.7 https://github.com/jqlang/jq/pull/2646

Great to hear and that was one of my hopes! but honestly it initially came to be because i was fiddling with some jq AST-tree stuff for fq :) weirdly it was much easier to implement than i expected. Hardest part was how to handle infix operators, +/- etc, parsing without infinite recision. But once i found and managed to implement precedence climbing things got a lot easier, it's still a bit of magic to me how well it works :) the eval part had some difficulties but mostly straight forward when you can piggy-back on the "host jq", but i tried to stay away from piggy-back too much, to not piggy-back at all probably requires implement a VM somehow.

BTW your very welcome to help improve the jq documentation. Me and some other maintainers have been talking about that it probably needs an overhaul to be more approachable and also better document some nice hidden features. Join the discord if you want!

Thanks for the jqjq shoutout! :) i'm quite sure jq is turing complete, jq (and jqjq!) can implement brainfuck https://github.com/01mf02/jaq/blob/main/examples/bf.jq

I think it's quite nice and intuitive to describe a pipe of filters as: filter | filter | filter. Also note that a single filter in jq is 1:N, 1 input N output where N can be zero which is a very useful feature. To express this in python etc you would probably need nested for i in (for j in ...)) and/or also use nested yield from somehow which results in a note a very CLI/ad-hoc query friendly syntax

Can recommend the documentary It's Quieter in the Twilight https://www.imdb.com/title/tt17658964/ that follows the engineers working with voyager 1 and 2

Some joke/useless software i've written:

- https://github.com/wader/jqjq

- https://github.com/wader/flac.tcl

In most cases i learned way more then i expected. Even made some friends.

yeap that could be an extension of the shorthand syntax, but note that pick/1 also works with any "path expression" ex: pick(if .a then .b.c else .x end)

If the task was the produce an object with the top 10 commiters with username as key and value number of commits maybe something liket this:

    group_by(.commit.author.name) | map({key: first.commit.author.name, value: length}) | sort_by(.value) | reverse[:10] | from_entries

jq support regex so now you can mix!

I do lots of exploratory work in various structure data, in my case often debugging media filea via https://github.com/wader/fq, which mean doing lots of use-once-queries on the command line or REPL. In those cases jq line-friendly and composable syntax and generators really shine.

jq has regex support! :)

Nothing planned but there has been some talks about it. https://github.com/jqlang/jq/issues/2855 is probably the lastest issue mentioning it

If you don't need to digg deep it's also possible to do:

  $ jq -n '{"a": 1, "b": {"c": 2, "d": 3}, "e": 4} | {a, e}'  
  {
    "a": 1,
    "e": 4
  }