Comment by lifthrasiir | Hacker News Reader

Comment by lifthrasiir | Hacker News Reader

Parent Full thread

> I think the requirement for Unicode is bad in general, whether it uses BOM or not.

While there exist legitimate complaints about Unicode, I believe there is no other feasible encoding than UTF-8 for textual formats now.

> I assume it means that numbers are expected to be IEEE 64-bit floating point numbers represented in the decimal format.

I too believe so (hence the next item), but that just doesn't make sense if you ponder. IEEE 754 doesn't define any textual format while it does have binary decimal formats. The correct wording should have been that numeric scalars follow a specific grammar to be interpreted as an IEEE 754 binary64 number in the data model.

> I think that a binary format would be better as a canonical representation anyways. However, the lack of indicating the data type would seem to make it difficult to know how to convert it unless you already know the schema.

That's fair. But we have already observed textual formats being... "abused" for the cryptographic purpose (e.g. JWT), so it's not too bad either to have a canonical representation or to explain why there is no canonical representation defined.

> Xenon does have a reference type to other nodes, which is something that other similar formats don't have, though.

There are several formats that do try to support native graph types, including YAML and Concise [1]. So that is hardly new. I think Concise actually tried very hard to make it fine! But it became quite more complex as a result.

[1] https://concise-encoding.org/

> I believe there is no other feasible encoding than UTF-8 for textual formats now.

I disagree, although I do not believe that this disagreement should have to affect some file formats, since some file formats should not need to care about the character encoding, except perhaps specific details, e.g. that ASCII bytes have ASCII meaning and non-ASCII bytes have non-ASCII meaning (which is true of UTF-8 and of some others).

> IEEE 754 doesn't define any textual format while it does have binary decimal formats. The correct wording should have been that numeric scalars follow a specific grammar to be interpreted as an IEEE 754 binary64 number in the data model.

OK. Although it seems clearly enough, probably the document should specify explicitly its working anyways, like you mention.

> There are several formats that do try to support native graph types, including YAML and Concise [1]. So that is hardly new.

I had seen Concise as well, and I have criticisms of that too. I did forget it though as I was writing my message.

I did have a idea to add a reference type too (you can see my other comment about TER and ASN.1X), although I would have it to not have the same meaning as the data it references (therefore meaning that it cannot result in cyclic graphs, even if it references itself). I think I should not have remote references, though. Concise encoding seems to have it have the same meaning as the data it references, and I think it is useful to not do this (you could use compression if you want to deal with many of duplicated data).

> I disagree, although I do not believe that this disagreement should have to affect some file formats, [...]

Ah, if you just want to make the UTF-8 requirement less strict for simpler decoders, that can be actually okay. My belief is more about legacy encodings like Shift_JIS.

I never seriously tried to fit any graph struture into a serialization format, so I don't yet have a very concrete opinion besides from what was already said. (I'm not even sure Concise had a good approach for that...)

That is hard to believe, the data being serialized may always have a graph structure.

"May", not "should", so some data don't have a graph structure. It really boils down to the ratio between non-graph data and graph data. I have some reason to believe that the former far outnumbers the latter, but maybe you have some concrete evidences against?

You misunderstand. Given that data MAY be a graph structure, the markup MUST support that. A tree with pointers to the parent nodes has a graph structure, a common enough occurrence.

Data MAY also include Egyptian hieroglyphs not encoded in Unicode. Does it mean that the markup MUST support them as well? I bet not. No format can natively support every possible data structure; serialization formats only have to support enough of them to which remaining structures can be isomorphically mapped.

> a common enough occurrence

Do you have a concrete figure about "common enough"? Otherwise we are only talking through anecdotal evidences (not necessarily bad though!). At least in my experience, graph structures do occur from time to time but almost all of them can be easily rewritten to non-graph form because they are typically DAGs and obviously ranked (i.e. its preferred topological order forms partitions of different nodes). The general graph that requires something like references was very rare.

The point is one can apply the xᴇɴᴏɴ library to arbitrary data and generate representative markup. For “Egyptian hieroglyphs not encoded in Unicode” 𓀠 one can write plug-ins.

Reply on news.ycombinator.com