on the other hand most database decisions are about finding the sweet spot compromise tailored toward the common use case they are aiming for, but your comment sound like you are expecting a magic trick.
Sticking data into the keys is definitely a thing I've seen.
One I've done personally is dump large portions of a Redis DB into a JSON object. I could guarantee for my use case it would fit into the relevant memory and resource constraints but I would also have been able to guarantee it would exceed 64K keys by over an order of magnitude. "Best practices" didn't matter to me because this wasn't an API call result or something.
There are other things like this you'll find in the wild. Certainly some sort of "keyed by user" dump value is not unheard of and you can easily have more than 64K users, and there's nothing a priori wrong with that. It may be a bad solution for some specific reason, and I think it often is, but it is not automatically a priori wrong. I've written streaming support for both directions, so while JSON may not be optimal it is not necessarily a guarantee of badness. Plus with the computers we have nowadays sometimes "just deserialize the 1GB of JSON into RAM" is a perfectly valid solution for some case. You don't want to do that a thousand times per second, but not every problem is a "thousand times per second" problem.
FoundationDB makes extensive use of this pattern, sometimes with no data on the key at all.
SICK: Streams of Independent Constant Keys
And "maps" seems to be a use case it is deliberately not aiming at.
You could store this as two columnar arrays but that is annoying and hardly anyone does that.
You would do a query like "give me all users with age over 18" or something and return a `{ [id: string]: User }`
If that optimization isn't for you, choose a different library.
If that optimization works for your use-case, it can make a huge difference.
The whole point of this project is to handle efficiently parsing "huge" JSON documents. If 65K keys is considered outrageously large, surely you can make do with a regular JSON parser.
You can split it yourself. If you can't, replace Shorts with Ints in the implementation and it would just work, but I would be very happy to know your usecase.
Just bumping the pointer size to cover relatively rare usecases is wasteful. It can be partially mitigated with more tags and tricks, but it still would be wasteful. A tiny chunking layer is easy to implement and I don't see any downsides in that.
Presumably 4 bytes dedicated to the keys would be dwarfed by any strings thrown into the dataset.
Regardless, other than complexity, would there be any reason to not support a dynamic key size? You could dedicate the first 2 bits on the key to the length of the key. 1 byte would work if there's only 64 keys, 2 bytes would give you 16k keys and 3 4M. And if you wanted to you could use a frequency table to order the pointers such that more frequently used keys are smaller values in the dictionary.
What is the difference?
The limitation comes with benefits.
I was just responding to the “X is an absurd way to do JSON”. Which seemed to single out objects vs arrays.
Like in this case maybe, but I don’t see a reason to make that general statement.