A fast EDN (Extensible Data Notation) reader written in C11 with SIMD boost

Huh. I haven’t run into this, although I totally see the problem. It’s backdooring types into JSON that it doesn’t support. I agree JSON’s number types are weak; it’s been a source of real problems for me. Given that observation, you can go in two directions: have richer types, like EDN, or give up on types in JSON entirely, which is the alternative I’d propose. I need to put my money where my mouth is here and implement something to demonstrate what I’m talking about, but imagine if JSON didn’t have numbers at all. The receiver would have to convert values to numbers after decoding, but I’m arguing that’s fine because in practice you have to check the value’s type anyway before you can use it.

When I say “syntax typing,” I mean that, for example, 31 is a number and “blue” is a string, and we know that because the string has quotation marks around it and the number is made of decimal digits.

This question has been answered by someone else, but I have my own comments about this as well so I will write it also.

EDN does complicate the syntax (so does XML, TER, various extensions of JSON, etc), but DER (and SDSER, if you want streaming) avoids this problem because the framing is the same for all data types, even though it has many different types and the encoding of the values of each type.

> That [the whole world seems to run on JSON] is true, and I don't like that :)

I agree with you (well, not everything but too many things); I don't like that either.

> From my perspective JSON syntax is too "light" and that translates to many complications typically in the form of convention

I agree with you about that too. In this case it is a number (there are problems with the numeric types in JSON), but there is also such things as: octet strings, date/time, non-Unicode text, etc.

> I agree JSON’s number types are weak; it’s been a source of real problems for me. Given that observation, you can go in two directions: have richer types, like EDN, or give up on types in JSON entirely, which is the alternative I’d propose.

Not only the number type (which is floating point so there is not a proper 64-bit or larger integer type, even though a integer type has been added into JavaScript after JSON was invented); the string type is also weak (since it cannot be arbitrary bytes), and so is the key/value list type (keys are only allowed to be strings and cannot be other types).

There are other directions as well; I think DER does in between because of the same framing for all types even though there are many types (you do not have to use all of the types; it seems that some people don't like it apparently due to the expectation that you have to use all of the types, but that is wrong). (DER also has the advantage of the canonical form if you need it (DER is already canonical form); although there is a canonical form for JSON, it is a bit messy, and apparently the canonical form for numbers in JSON is complicated.)

If you want to give up on types entirely, then why should you use JSON?

Specifically, I refer to what is quoted below:

> Upon encountering a tag, the reader will first read the next element (which may itself be or comprise other tagged elements), then pass the result to the corresponding handler for further interpretation, and the result of the handler will be the data value yielded by the tag + tagged element, i.e. reading a tag and tagged element yields one value.

> If a reader encounters a tag for which no handler is registered, the implementation can either report an error, call a designated 'unknown element' handler, or create a well-known generic representation that contains both the tag and the tagged element, as it sees fit. Note that the non-error strategies allow for readers which are capable of reading any and all edn, in spite of being unaware of the details of any extensions present.

Due to these things, EDN does not have a proper "octet string" type, even if the extension is added.

> This implementation does not do streaming for now, but it understands a concept of "reading one complete" element from buffer. The only missing part is buffer managment.

OK, then it could be improved.

> Could you explain how it is restricted if you are allowed to run whatever you want during reading of edn document? You can even do IO, no restrictions at all!

Perhaps the above explains how it is restricted. It does not prevent you from looking up data in a database, etc; it is the data model of EDN itself which is restricted; it is not restricting what you do with it.

> Please show how it does better. I'm very curious

Since all types using the same framing, you can do "lazy decoding" if appropriate (you can also use custom decoders in any part of the file, and this can depend on the schema), and ASN.1 does have a built-in octet string type (as well as bit string, unrestricted character string, etc), and you can add implicit or explicit tagging (I prefer to use implicit if the underlying type is sequence or octet string, and explicit otherwise), as well as types such as External (and the nonstandard ASN1_IDENTIFIED_DATA type), you can easily define any type and can easily skip past any field of any type.

> #init/postgres {:db-spec {:host "..." :port 54321 ,,,} :specs {:user ,,,}} [#user/id 1 #user/id 2 #user/id 3]

Even with TER (the below does not use any extensions to TER itself, but extensions to TER are also possible; even if not, whoever reads the resulting DER can handle the application-specific types as needed), you can:

  [ [P:(database.example) 54321] [0A:1 0A:2 0A:3] ]

  [ [P:(database.example) 54321] [0A[1] 0A[2] 0A[3]] ]

(TER is not one of the official ASN.1 formats; it is one that I invented for the purpose of having a text format for ASN.1 which can then be converted to DER; most programs would be expected to use DER rather than TER.)