Hacker News Reader

- The flexibility in name is to support the plethora of programming languages.

- Commas are for readability as per English.

- Three digits are recommended for the encoder.

- Textual data is decoded into binary information in software so setting expectations, e.g. supporting ∞, facilitates interoperability by reducing impedance mismatches.

- If the implementation uses ɪᴇᴇᴇ doubles the rounding shall be as expected.

- Unicode surrogates are disallowed.

- We are following the Base64 standard exactly.

- It is essential for data to be structured as a graph, it simply occurs. Serialization on most data formats has the required support for graphs awkwardly layered on top of the encoding.

- The C# implementation defines the data model; essentially objects, array and scalars with multiple parents for nodes. The Formats section is to, as stated, facilitate interoperability.

- Limiting whitespace to spaces and tabs does speed processing. The ᴜᴛꜰ-8 bytes can be left in the input buffer rather than copied. All significant bytes are ᴀꜱᴄɪɪ.

- There are no unmatched angle brackets (that was a typo).

- I know the reasoning behind XML.

> The flexibility in name is to support the plethora of programming languages.

As far as aware, there is no language that allows a tab in its identifier. I'm aware of some (uncommon) languages that allow a space in identifiers though.

After all, the goal of language is firstly to set what's valid or not and secondly to give a meaning to valid one. The valid set should be as maximally different from the invalid set as possible, but allowing virtually invalid characters in names is against this goal. Consider the approach taken by XML 1.1 (not 1.0) if you need an idea.

> Commas are for readability as per English.

A lot of English-speaking countries actually have `.` and `,` swapped: 3,141.592 vs. 3.141,592 for example. Due to this ambiguity, a comma as a grouping separator is heavily discouraged. You can instead use a underscore `_` (very common in programming languages) or a space (more preferred in human texts) without such concerns.

> Three digits are recommended for the encoder.

Also, some English-speaking countries use different group sizes (notably India). I'm personally fine with three-digit groupings despite of that, but that should be clearly specified at the very least.

> Textual data is decoded into binary information in software so setting expectations, e.g. supporting ∞, facilitates interoperability by reducing impedance mismatches.

This procedure should have been made explicit. In fact, the single worst thing you can do in serialization formats is an unclear definition of data model.

> If the implementation uses ɪᴇᴇᴇ doubles the rounding shall be as expected.

There is nothing like "as expected" in IEEE 754. The current rounding mode is a part of the execution state, so leaving it unspecified risks a non-deterministic interpretation even in the same execution. Either you should specify some rounding mode (most likely round-to-even), or you should state that encoders should pick a long enough decimal representation to avoid any such issue.

> We are following the Base64 standard exactly.

Which base64 standard in [1]? The padding in base64 is vestigial anyway and serves no practical need by now, so there is no strong reason to use a particular standard with the required padding. It is much more important to decide what to do with incorrect padded bits.

[1] https://en.wikipedia.org/wiki/Base64#Variants_summary_table

> It is essential for data to be structured as a graph, it simply occurs. Serialization on most data formats has the required support for graphs awkwardly layered on top of the encoding.

I don't question that graph structures often occur naturally and existing schemes are often awkward, but I think that's more of the lack of co-developed standards. I have outlined my rationale for layering in other comments.

> The C# implementation defines the data model; essentially objects, array and scalars with multiple parents for nodes. The Formats section is to, as stated, facilitate interoperability.

The data model should be abstract enough to be truly interoperable. JSON suffered a lot from having no defined data model to this day, even when there was a soft-of-reference implementation by Crockford. It is not too hard to define a data model in prose rather than code.

> Limiting whitespace to spaces and tabs does speed processing. The ᴜᴛꜰ-8 bytes can be left in the input buffer rather than copied. All significant bytes are ᴀꜱᴄɪɪ.

May have been true in the past, but it's no longer true since SIMD-based parsers. Also the very existence of escape sequence prevents the true non-destructive zero-copy (aka in-situ) parsing anyway. With such sequences, zero-copy/in-situ parsing has to be destructive to be performant and that can preclude some use cases. Allowing additional space characters is much easier than that.

> There are no unmatched angle brackets (that was a typo).

I meant to refer to `<<Name>...<<$>`. Multiple grouping characters can allow for simpler syntaxes.

> Consider the approach taken by XML 1.1

I evaluated xᴍʟ 1.0 and 1.1’s restrictions but consider the alternative, accepting anything but the empty string to be simpler.

>A lot of English-speaking countries actually have `.` and `,` swapped:

Which?

>...should state that encoders should pick a long enough decimal representation to avoid any such issue.

.net for example has round trip encoding to achieve this.

> Which base64 standard in [1]?

As linked in the document, we refer to ʀꜰᴄ 4648 for Base64 using the simplest version which states that “Implementations MUST include appropriate pad characters at the end of encoded data unless the specification referring to this document explicitly states otherwise”.

> I have outlined my rationale for layering in other comments.

>> Graph is generally a bad thing to encode at this level

The ᴅᴏꜱ attach you are concern about is not applicable. Layering graphs on top of the encoding in a separate standard is not practical. As stated serialization requires a graph structure.

>It is not too hard to define a data model in prose rather than code.

I have done that.

>May have been true in the past, but it's no longer true since SIMD-based parsers. Also the very existence of escape sequence prevents the true non-destructive zero-copy (aka in-situ) parsing anyway. With such sequences, zero-copy/in-situ parsing has to be destructive to be performant and that can preclude some use cases. Allowing additional space characters is much easier than that.

SMID based parsers would almost certainly run faster with simplified byte handling rules. Practically, when say skipping whitespace, searching for two specific bytes is faster than the alternatives. Most strings do not have escape sequences so do not need to be copied, a scan for '\' is fast.

> `<<Name>...<<$>`

This is to be terse. An array type in an xᴍʟ like language is a leap forward. Additional significant characters would over complicate the standard.

>>A lot of English-speaking countries actually have `.` and `,` swapped:

>Which?

Here's a table https://en.wikipedia.org/wiki/Decimal_separator#Examples_of_...

EDIT: Ah, now that I re-read that, I see the "English-speaking" specifier.

> Which?

Oh, I just realized that I carelessly put "English-speaking" there. (AFAIK the exact reversal does exist in English, but is much rarer and not entirely domestic.) But that doesn't really justify the use of comma in numbers.

> .net for example has round trip encoding to achieve this.

That was never guaranteed to my knowledge, and there also seems an apparent difference in .NET Framework and .NET Core/Runtime versions according to some searches. Many enough standardized languages have no strong requirement as well (for example, see [1] for ECMAScript), so this should be clearly specified to be truly portable.

[1] https://tc39.es/ecma262/multipage/abstract-operations.html#s...

> As linked in the document, we refer to ʀꜰᴄ 4648 for Base64 using the simplest version [...].

You are free to rephrase any variant of base64 standard into simple statements. (Ideally the standard itself should be also cited, though.) I was asking why that particular variant was used.

> Layering graphs on top of the encoding in a separate standard is not practical. As stated serialization requires a graph structure.

The separateness here is not binary. The Unicode standard (ISO/IEC 10646) for example has multiple standard annexes [2] that are synchronized with the core standard but published and developed separately. For all practical purposes they form a single unified standard, but you can (and almost likely should) ignore some irrelevant annexes. I'm arguing for a similar structure to be clear, do you think that's also a no-go?

[2] https://www.unicode.org/reports/

> I have done that.

The only explicit thing in your data model is the following definition:

    // Read as: "document" is either "object", "array" or "scalar".
    document = object | array | scalar
    // Read as: "object" is a distinct type "Object" with "name" and zero or more "field"s. (And so on)
    object = Object(name, field*)
    array = Array(name, item*)
    scalar = Scalar(name, value)

This is not what's implied by following sections. To begin with, you have no explicit definition of `item` or `field`. Yes, it is kinda obvious that both should be `document` and also can be somehow inferred that `scalar` in the `item` place should be a nameless object, but not only such relations are not explicit but now we have another class of objects not mentioned in the first place! The exact nature of `value` is also very unclear.

In my understanding, your data model is more like this:

    document = object | array | scalar

    // `...?` for zero or one copy of the preceding data.
    optional-name = Name(name)?
    optional-ref-id = RefId(ref-id)?
    optional-type = Type(type)?

    name = string             // Can't be empty
    ref-id = string           // Globally unique
    type = string             // Interpretation up to clients

    object = Object(optional-name, optional-ref-id, optional-type, field*)
    field = ObjectField(name, optional-ref-id, optional-type, field*)
          | scalar

    array = Array(optional-name, item*)
    item = object
         | ScalarItem(optional-ref-id, field*)  // Implicitly converted to object
         | ReferenceItem(reference-id)

    scalar = Scalar(name, value)
           | Reference(name, ref-id)
    value = string    // Additional interpretation rules may exist after parsing

For example, the following document:

    <Person>
        <Name=Bonnie>
        <Spouse#jack-smith>
            <Name=Jack>
        <$>
        <Doctor=@jack-smith>
    <$>

...is thought to parsed into the following underlying data:

    Object(Name("Person"),
        Scalar(Name("Name"), "Bonnie"),
        ObjectField(Name("Spouse"), RefId("jack-smith"),
            Scalar(Name("Name"), "Jack")
        ),
        Reference(Name("Doctor"), "jack-smith")
    )

This abstraction clearly demonstrates many important things coming up in implementations. For example, every `field` has to provide `name` because all possible branches (`ObjectField`, `Scalar` or `Reference`) have one---at least in my understanding. I shouldn't be figuring out such data model myself to be honest; it's your job to provide one.

>But that doesn't really justify the use of comma in numbers.

Readability.

>I was asking why that particular variant [of Base64] was used.

I repeat, we are using the ʀꜰᴄ 4648 variant.

> [separate standards documents] do you think that's also a no-go? Yes.

Data model here:

document := named-entity

named-entity := named-object | named-array | named-scalar

named-object := NamedObject(name, object)

named-array := NamedArray(name, array)

named-scalar := NamedScalar(name, scalar)

object := Object(id?, type?, field*)

array := Array(id?, type?, item*)

scalar := Scalar(id?, type?, value)

field := named-entity

item := object | array | scalar

name := CHAR+

id := CHAR+

type := CHAR+

value := CHAR*

> Readability.

Okay, I see you demand a very explicit question: why did you use comma instead of other characters for grouping?

> [separate standards documents] do you think that's also a no-go? Yes.

I like to hear why then, given this approach seems to be very successful for Unicode and many others.

> Data model here:

Thank you for the grammar. The document will massively benefit from such explicit description, I completely missed an unnamed scalar for example (because it has no syntax at all). The grammer doesn't have to be reproduced at all, for example the following is my attempt to rephrase it in prose:

There are three kinds of entities in Xenon: objects, arrays and scalars. All entities may optionally have a name, a type, an ID or any combination of them. Objects can contain named entities or fields, where field names should be distinct within the same object. Arrays can contain either named or unnamed entities. Scalars can contain zero or more characters. Names and types are non-empty free-form strings and no meanings are assigned for them, but this document gives a guideline for common types to ensure the interoperability. IDs should be distinct within the same document, but otherwise they are only used to encode a graph structure and have no other meaning. Objects and arrays can contain a reference which can stand in for any entity with given ID; references in an object needs its own name which can differ from the referent.

Please consider a similar clarification early in the document.

> why did you use comma instead of other characters for grouping?

Because English is the global lingua franca

> I like to hear why then

Unlike Unicode xᴇɴᴏɴ must be defined in one grammar. Having special names, like $type, and hoping no language uses them is fragile.

> Because English is the global lingua franca

But English is not the largest language in terms of the number of speakers. I speak Korean for example and three-digit grouping is unnatural in Korean, which uses myriads instead. I guess Xenon is designed to be not comfortable for the vast majority of people including me then?

> Having special names, like $type, and hoping no language uses them is fragile.

I was not proposing special names after all. Xenon clearly has a much better extension point, namely the type string. You can have just one reserved character, which is very unlikely to appear in verbatim, and use it for the extension. And it is even unclear whether JSON's use of `$` for such special names was fragile after all, given that such complaint seems uncommon. (I complained about that in the past, but I mean others seemed to be cool.)

In addition, I don't think you have mapped enough languages to conclude so anyway. For example, can you list all popular enough languages that allow `$` in identifiers? While whether to allow `$` or not requires a single example, maybe you have missed other identifiers in other languages! (Angle brackets are also common in stringified types, after all.) So I expected you to have at least looked at them for your conclusion.

> I guess Xenon is designed to be not comfortable for the vast majority of people including me then?

You seem fairly comfortable with the rest of English. Korean numbers do not seem to scale well.

> allow `$` or not requires a single example

Java allows $ in identifiers.

I have silenced all criticism.

The new Xenon Design Rationale shows how Xenon is the best data description language.

https://xenondata.org/xenon-design-rationale.html

Comment at https://news.ycombinator.com/item?id=42178359

> The requirement for the mandatory BOM is unacceptable for most non-Windows users

I think the requirement for Unicode is bad in general, whether it uses BOM or not.

> What on earth is "IEEE 64 bit double precision floating point numbers" in the context of textual format?

I assume it means that numbers are expected to be IEEE 64-bit floating point numbers represented in the decimal format.

> If you have to escape the base64 padding, maybe you should just drop the padding?

I agree.

> While technically a matter of choice, it uses a lot of unmatching angle brackets and that's just... ugly.

I agree with that too.

> No canonical representation.

I think that a binary format would be better as a canonical representation anyways. However, the lack of indicating the data type would seem to make it difficult to know how to convert it unless you already know the schema.

> Maybe you should review tons of other alternatives in this space

Just a few days ago, and still working on it today, I had been making something called TER. It might also be worth to look, and I wrote another comment relating to this.

Xenon does have a reference type to other nodes, which is something that other similar formats don't have, though.

> I think the requirement for Unicode is bad in general, whether it uses BOM or not.

While there exist legitimate complaints about Unicode, I believe there is no other feasible encoding than UTF-8 for textual formats now.

> I assume it means that numbers are expected to be IEEE 64-bit floating point numbers represented in the decimal format.

I too believe so (hence the next item), but that just doesn't make sense if you ponder. IEEE 754 doesn't define any textual format while it does have binary decimal formats. The correct wording should have been that numeric scalars follow a specific grammar to be interpreted as an IEEE 754 binary64 number in the data model.

> I think that a binary format would be better as a canonical representation anyways. However, the lack of indicating the data type would seem to make it difficult to know how to convert it unless you already know the schema.

That's fair. But we have already observed textual formats being... "abused" for the cryptographic purpose (e.g. JWT), so it's not too bad either to have a canonical representation or to explain why there is no canonical representation defined.

> Xenon does have a reference type to other nodes, which is something that other similar formats don't have, though.

There are several formats that do try to support native graph types, including YAML and Concise [1]. So that is hardly new. I think Concise actually tried very hard to make it fine! But it became quite more complex as a result.

[1] https://concise-encoding.org/

> I believe there is no other feasible encoding than UTF-8 for textual formats now.

I disagree, although I do not believe that this disagreement should have to affect some file formats, since some file formats should not need to care about the character encoding, except perhaps specific details, e.g. that ASCII bytes have ASCII meaning and non-ASCII bytes have non-ASCII meaning (which is true of UTF-8 and of some others).

> IEEE 754 doesn't define any textual format while it does have binary decimal formats. The correct wording should have been that numeric scalars follow a specific grammar to be interpreted as an IEEE 754 binary64 number in the data model.

OK. Although it seems clearly enough, probably the document should specify explicitly its working anyways, like you mention.

> There are several formats that do try to support native graph types, including YAML and Concise [1]. So that is hardly new.

I had seen Concise as well, and I have criticisms of that too. I did forget it though as I was writing my message.

I did have a idea to add a reference type too (you can see my other comment about TER and ASN.1X), although I would have it to not have the same meaning as the data it references (therefore meaning that it cannot result in cyclic graphs, even if it references itself). I think I should not have remote references, though. Concise encoding seems to have it have the same meaning as the data it references, and I think it is useful to not do this (you could use compression if you want to deal with many of duplicated data).

> I disagree, although I do not believe that this disagreement should have to affect some file formats, [...]

Ah, if you just want to make the UTF-8 requirement less strict for simpler decoders, that can be actually okay. My belief is more about legacy encodings like Shift_JIS.

I never seriously tried to fit any graph struture into a serialization format, so I don't yet have a very concrete opinion besides from what was already said. (I'm not even sure Concise had a good approach for that...)

That is hard to believe, the data being serialized may always have a graph structure.

"May", not "should", so some data don't have a graph structure. It really boils down to the ratio between non-graph data and graph data. I have some reason to believe that the former far outnumbers the latter, but maybe you have some concrete evidences against?

You misunderstand. Given that data MAY be a graph structure, the markup MUST support that. A tree with pointers to the parent nodes has a graph structure, a common enough occurrence.

Data MAY also include Egyptian hieroglyphs not encoded in Unicode. Does it mean that the markup MUST support them as well? I bet not. No format can natively support every possible data structure; serialization formats only have to support enough of them to which remaining structures can be isomorphically mapped.

> a common enough occurrence

Do you have a concrete figure about "common enough"? Otherwise we are only talking through anecdotal evidences (not necessarily bad though!). At least in my experience, graph structures do occur from time to time but almost all of them can be easily rewritten to non-graph form because they are typically DAGs and obviously ranked (i.e. its preferred topological order forms partitions of different nodes). The general graph that requires something like references was very rare.

The point is one can apply the xᴇɴᴏɴ library to arbitrary data and generate representative markup. For “Egyptian hieroglyphs not encoded in Unicode” 𓀠 one can write plug-ins.

Thanks for writing this up! Sounds like you really dig deeply in to this.

Not entirely sure what you mean with canonical representation (I've heard this in the context of JSON-LD before, though). Can you explain what you mean here?

Where do you see the problem with Graphs and Dos? A reference is just a pointer. You just have to be careful when doing recursive code. I actually like the idea to explicitly define how a reference / association is made, because otherwise people will have to re-invent ID and association concepts and there's no shared understanding. In JSON Schema, you cannot properly express an association or graph structure and people start using overloaded and not well-defined concepts like `$ref` which is a separate standard.

Here the canonical representation refers to one single definite and unambiguous encoding for given data. This requirement is very common in cryptographic applications and also commonly demanded when the deterministic processing is desired. Technically the "canonical" and "deterministic" encoding can differ (e.g. ASN.1 CER vs. DER), but there is not much value to have two distinct encodings.

On graphs: as you've said recursive code has to be careful, but the recursion itself is not very frequent in normal applications and they are more susceptible to attacks if the recursion is built into the serialization format. XML billion laughs attack is a famous demonstration for this issue. It is still worthwhile to have parallel standards to specify how recursive structures should be encoded in the basic format, and possibly to tweak the basic format to better accommodate such standards, but I believe such needs can be met without making the basic format bigger.

Is canonilization not irrelevant. Any format with comments is not canonical; so xᴍʟ is not, ᴊꜱᴏɴ has escapes options for string characters so it not either.

Response to graph conjecture stated https://news.ycombinator.com/item?id=42072133.

Almost no serialization format is canonical by default---AFAIK bencode was the sole example that mandates the canonicalization. Instead, a canonical subset of the format is usually defined, which would of course exclude comments, unless comments themselves are considered semantic like XML. Yes, even XML has a canonical subset [1]!

[1] https://www.w3.org/TR/xml-c14n/

There is the concept of “well formed xᴇɴᴏɴ” as outputted by the ᴅᴏᴍ and serializer which is deterministic as will suffice for canonicalization.

There is no mandatory bom, it's recommended, not a must

Thank you for noticing that, my mistake. Fixed by now.