We see this kind of Nete’s data all over the place (json, yaml, python dictionaries, toml, etc, etc) and I’m thinking wouldn’t it be nice if we had a path language that worked across these structures, just like how we can regex any strings?
So we can have a pathql executable that we can feed yaml and json data to, but I can reuse the query in Python when I want to extract values from a json stream I just deserialized.
I think you could well apply jsonpath to yaml, except for the different data types, which is what makes you need xmlpath, jsonpath, file path, css and so on. If you're willing to do some automagic conversions, you could probably do that right now, if you write the code for it.
They vary in syntax and how they deal with scalars, objects and collections. Having to write 'myKey' is a bit unfortunate in json for example. Still, for any document larger than 20 lines yaml will fall apart easily.
Xml (dialects) mark the beginning and end of nodes explicitly, which deviates from json/yaml. Xml can represent nodes within the value, which is impossible for json/yaml. To convert such xml value to the latter, you have to break up such a node in fragments and represent them as a collection in json/yaml.
Are you talking about XML like `<text>Something something <para> inside </para> something else</text>`? I thought this would also presented as the text element having three children, the text "Something something", the <para> tag with its subtree, and the text "something else". Am I misremembering?
It's just when writing the document that you get the advantage, the data model is the same.
I think that is one reason why json took of for exchanging machine readable data, xml is too expressive, and leans towards document authoring.
Json got introduced because XMLHttpRequest was an IE-only component. Data exchange was done mostly between servers, back then the front-end was really dumb. Why xml got slowly replaced is because that for machine generated data, json proved to be sufficient. Any json library is easier to work with then their xml counterpart, exactly because of what we discuss here.
(Looking at you, jq, though to be fair JSON itself supports null...)
If you want this kind of semantics with JSON, take a look at https://www.jsoniq.org
But one problem is, that each format has slightly different ways it works. Some have nodes which support properties, some not. This makes building a proper query-language a bit more complicated if it should not end up ugly.
Yes, if you want to make the query language reject as incorrect queries that specify a property on a node that can't have properties, it is messy. But is this really necessary? You can write faulty queries anyway. And it's not a programming language for complex systems, where it really helps to prevent mistakes early. SQL works fine with no type safety and so on.
Speaking of this topic, I would be very interested in a CLI tool that translated between different dialects of regex. As opposed to general data or configuration, I find the case for cross format conversion very compelling for regexes; the objects that different regex languages deal with are functionally completely identical. I would be very happy to be able to craft a regex to search for something in vim, then convert the regex to grep or use another tool on the shell, then perhaps adapt it to some other scripting languages e.g. Javascript/Ruby/Perl.
Indeed I am sad that I am not finding any CLI tools (non web-based) for this beyond https://github.com/Anadian/regex-translator . Am hopeful for more suggestions.
torepr didn't quite work for me as I was dealing with objects containing large binary blobs and it was awkward.
fq is a great tool and I shouldn't have suggested this was a problem unique to it! I think this kind of "issue" is inevitable when dealing with so many types of input. And to be honest I struggle hard using jq as well for anything other than very basic paths, due to infrequent usage.
About blobs, if you want to change how (possibly large) binaries are represented as JSON you can use the bits_format options, see https://github.com/wader/fq/blob/master/doc/usage.md#options, so fq -o bits_format=md5 torepr ...
I can highly recommend to learn jq, it's what makes fq really useful, and as a bonus you will learn jq in general! :)
- Integers (including 64-bit integers and longer)
- Non-finite floating point (Infinity, NaN)
- Keys of types other than strings
- Non-Unicode strings (e.g. byte sequences, TRON code, etc)
- Date/time
- Links
Additionally, they may differ of whether or not the order of keys should be retained.
You generally don't see yaml or XML be called that, but there's some information on the net that you can find about it.
object graph - grove plus object references. eg serialization formats
knowledge graph - object graph where parent-child relations are explicit, using subject-verb-object clauses. like how Java Spring "flattens" object graphs.
document - grove plus random text nodes