Show HN: Weird JSON superset with BigInt, Infinity, TypedArray etc | Hacker News Reader

Show HN: Weird JSON superset with BigInt, Infinity, TypedArray etc

github.com

67 pointsgraderjs39 comments

Sorry, but this is awesome. It's not really what you probably want in most situations. But sometimes you have constraints that come with legacy systems. Adding another layer of Base64 encoding comes with its own set of compatibility quirks, impedes introspection and debugability, and still doesn't help with BigInt's and the like.

For systems where you have modern stacks and control all the moving parts, something like Cap'n Proto will likely make everyone happier in the end. But you don't always have that luxury, and this project is very cool in its own right...

I added this to my project, and it was at least 250% faster than standard JSON functions somehow. 10/10 would recommend.

I'm of the opinion that any types other than strings in JSON is a mistake - any parsing assumptions will _never_ be future sufficient. It should be the user of the library's responsibility to implement their own string->type conversions.

Even null and booleans?

I think null at least is a great example of their point since many applications might not want a "null" type.

Although I am not sure I would go as far as to drop booleans. As for numeric types, that's a complicated issue, I think they should be supported for pragmatic reasons even though it's impossible to come up with a numeric type that satisfies everyone.

I think booleans might even be the most problematic, as it's _really_ implementation specific. In C for example you might expect 1 == TRUE, but in other languages this is not the case.

The real problems come in when you handle unexpected cases - and there's no real answer as to whether you return true, false or an error. Personally I think the responsibility should be thrown back at the coder. E.g. I will sometimes do:

    if(!(jobj.get("key", "true").contains("f")))fun();

Where "true" is considered safe and default. Generally you need to make some effort to switch it off by making sure an "f" exists somewhere, but either case is fine.

Imagine for example that the value "true" turns on something potentially dangerous, you might want to guard against accidentally switching it on with:

    if(jobj.get("key", "false").equals("true"))fun();

So you for sure have to make sure the string is "true" to switch it on, nothing else.

> Even null and booleans?

I would say so yes.

Pick null for example, depending on your language of choice, you might choose to utilize: "null", "NULL", "", "undefined", "\0", etc.

The same with booleans, are we speaking "true", "True", "TRUE", "1", "0xFFFFFFFF", or anything that isn't a false is true, etc.

Really how you want to handle these situations is entirely up to you. Personally I prefer some functions like:

    string s = jobj.get(key, default)

The default value being used if a valid string could not be found. Your default could of course be null or equivalent (handle the error), or it could be some known safe value (the show must go on).

> Pick null for example, depending on your language of choice, you might choose to utilize: "null", "NULL", "", "undefined", "\0", etc.

That's an argument against your position not for. By having a type "null", there's no issue of interoperability between languages who write null in different ways.

To me the "NULL" of JSON is really the lack of a value. I think having a NULL value could introduce problems.

For example: Did you set `{"address":null}` or did you set `{"address":0x00000000}`? It's important to know the difference, do you need to throw an error or warning? If your NULL value maps to something that is also valid input, how are you supposed to know how to handle the case? If you removed the "address" key on the other hand, you can simply do some exists() functions check.

I think putting types into the JSON spec only serves to create confusion. I think the JSON libraries should offer common helper functions for doing these conversions (e.g. string -> int), but ultimately it's the responsibility of the coder.

If you encode everything in strings then you have to have a shared schema between producer and consumer and redundant isset fields. Otherwise you can't be sure to interpret values. Is "name": "Null" a missing value or someone's surname. Is "country": "NO" a falsey value or Norway?

I would think that you would use a lack of a string to encode a NULL value, e.g.:

    { "name":"null", "age":"44" }

    { "age":"44" }

In the first example, your get function would return "null" for "name" and in the second example it would return the value of default, which would be implementation specific.

That demonstrates the exact problem, whatever string you use to represent the null value EG "null" is now a keyword that needs to be enforced.

So "null" can no longer be someones username, which will cause weird edge cases...

> whatever string you use to represent the null value EG

> "null" is now a keyword that needs to be enforced.

So in the JSON itself, you literally remove the field. In another comment I make is the point that NULL itself could be considered a proper value, you're just pushing the problem down the line. The library making such a decision for you is what may ultimately trip you up.

Instead, you could have something like:

    string name = jobj.get("name", NULL)

So that it does return NULL in this case if a name hasn't been given.

But consider an example where you're loading some code into a VM's RAM:

    int location = toint(jobj.get("address"))

NULL in this case is a perfectly valid place to load a program in a VM, or was it that there was an error? Or is it that no address was given in the JSON configuration?

Instead, we can specify our own "NULL" value:

    int location = toint(jobj.get("address", "-1"))

So now you're asking to load something before the first position of the VM memory, which indicates an error.

I think we're in agreement, originally it was suggested to use a "null" string, instead of NULL as a value, which is what I said might cause weird error cases.

> So in the JSON itself, you literally remove the field.

That's a good point, but in some cases having a `null` is an easier option, as you only need to determine if the value is valid, not if it exists AND is valid (but I suppose you could say it's always valid if it exists... that's if you trust the endpoint to provide valid values though).

> But consider an example where you're loading some code into a VM's RAM

Allocating memory locations from dynamically created JSON values is bound to lead to many different errors...

I guess my main point is that JSON is inherently untyped and suited for messy data or user input. This messiness makes data validation on the consumer of the JSON necessary, which means the consumer has to check if the value is valid before using it. In most cases `null` is considered invalid but if the field isn't there, then it could be an error in the JSON provider.

I completely agree. I use strings for json front / back communication as much as possible. A string (or hash) is a unique metric. I have a real-life example where I was working with a junior C # backend developer who was creating code with a lot of errors, and we doubled our progress when we stopped discussing json types. Another good lesson when the frontend code of the code thinks "with its own head" and checks each json input parameter for safety and fewer errors, and strings have less opportunity to check and validate if (!!s) and clean i=parseInt(res.i), parseBool(..), parseObjA(..) looks like typed programming and newcomers to projects can learn the code better.

And still no specification for `datetime`.

I can't for the life of me understand why protocol specs omit this.

Because people think they want "time since epoch" but they really want ISO8601 because they'd like it to be readable, and oh, btw, durations would be nice, too.

That spec is quite large, and everyone wants different parts of it. And different languages have different and somewhat incompatible support for datetimes.

And, worse, people want different behavior for invalid datetimes.

So your parser either balloons in size and requirements and its test-suite or restricts it to a handful of cases, at which point for most users, your datetime type is useless.

This is why if you have a protocol like JSON that allows you to construct arbitrary types or simply parse strings, you let users figure out how to handle it.

Nah. The notion of a "time since arbitrary epoch" is pretty universal, even though it may have a variety of representations. Just pick one, it really doesn't matter which.

Actually JavaScript already did pick a representation - milliseconds since 1970. It would be easy and unambiguous to add { "foo": T1606410158672 } to the grammar.

And I think nobody does that in a protocol because you're adding extra complexity to get you little more than transmitting a number.

You get significantly more than transmitting a number.

- You get automatic conversion to/from a known datatype, Timestamp, not to an arbitrary number

- You get automatic checks on the validity of this datatype

- You can specify exact precision of this datatype (for example, you may want to represent nanoseconds, not milliseconds, since epoch)

> And different languages have different and somewhat incompatible support for datetimes.

The support for datetimes is irrelevant if the protocol spec specifies how to serialize a date. Converters for languages will appear or already exist.

Datetimes is one of the most common data types and yes, everyone ends up "constructing arbitrary types or simply parsing strings and letting users guess how to handle stuff". Remember .NET's handling of datetime? [1]

ISO 8601 was published in 1988. You'd think that protocols designed in the 21st century would at least know about it. You don't have to require all of it. A sensible expectation of date and datetime would suffice the vast majority of use cases.

[1] https://www.hanselman.com/blog/on-the-nightmare-that-is-json...

> The support for datetimes is irrelevant if the protocol spec specifies how to serialize a date.

Consider JSON and how numbers are handled right now.

Per the spec, it's an arbitrarily long decimal, and some libraries do deserialize it to a BigDecimal class.

But many libraries in a chain of processing will treat it as a double, often libraries outside your control.

So you can't rely on them preserving any accuracy beyond what a double can handle.

Likewise, if you have a complex date-time representation, the spec can require all kinds of detail, but in practice, some of that data will be thrown out.

And worse, they won't throw an error, they'll truncate that data quietly.

Me: Datetime is one of the most used types beside the primitive ones. Let protocols specify what how it has to be serialized. This will serve as a baseline against which libraries will be measured. It significantly reduces questions on how to represent datetime, and libraries handling this have a significantly reduced space for caveats and errors.

You: no-no-no, it won't work, let us make it significantly worse. Let us not specify this, and let everyone specify their own ad-hoc solutions with manual parsing and checks which will fail in every which way imaginable and let people serialise datetimes as `"\/Date(1320825600000-0800)\/"` for all I care because ... ?

Welcome to the datetime apocalypse

So you here know, this is not a joke, nor a fun exercise. It is a serious project, I made it. I did just have some fun with the README tho ;P ;) xx

I mean it's clearly a joke, but... Why not? A new standard, great idea! Just a reminder to everyone, try this out in production!

I'll take it over YAML.

I know these are all jokes, but this doesn't actually assume the role of YAML or JSON, which are focused around human-readable, human-editable formats. This would be way cooler, IMO, if it didn't encode into base36 since there are plenty of good tools for binary-serialized formats.

Damn. This is clearly just a fun exercise by OP, and most people here are judging it as if this is a serious project they want people to use in production or something.

If you're going to enhance JSON, you might consider how you could support comments in unencoded files at-rest on disk.

Like JSON5?

I realize this is not entirely serious, but if it were to be improved, I would like to see the ASCII safety features moved into a separate product from the extended JSON type features. It would be easier to take some "extended" JSON encoded in UTF-8 and just base-64 it if I needed the ASCII safety.

I'm not sure whether this is a joke? Is it just to mess with devs? Is this what video streaming websites' "DMCA crypto" websockets will use in future? Is this like jsfuck for JSON?

Why use a static alphabet and not a dynamic one, e.g. with a magic header hidden somewhere that automatically changes the alphabet once parsed for the following content, including other magic alphabet headers? /s

edit: Can my comment make it into testimonials, too, plz?

Hmm, interesting idea. Have you ever heard of Base64? Just curious why you chose to name it 46? And why JSON? There's specific definitions of 46, and JSON, and they don't say anything about this. Did I miss something in the JSON spec?

Haters gonna hate, go for it!

Amazing job! JSON can be improved like HTML 1.0

Just commenting to get featured in the README

I don’t see the comment from HN here. Lies

This sucks. Real JSON is waay better than OP. Or at least use Protobuf, it's popular and created by Google, or bson. Don't use this, it's clearly a security nightmare.

Reply on news.ycombinator.com