I'm not sure what nomenclature I could use besides "data format" to describe what this is, any more than one could describe JSON as anything other than a data format...
The security aspect is where I'm having the most trouble communicating, because it's just not a well known issue. The problem comes from the different ways that codecs deal with the data they receive (for example Java Codec A vs Golang codec B vs Python codec C vs Python codec D, etc). If the spec isn't strict enough, a conformant codec could behave in ways that are insecure when coupled with other systems.
Security really does start at the specification, in that the spec has to define what a codec can and cannot do. For example:
"A codec MUST reject invalid characters. It MUST NOT truncate them or replace them with the Unicode replacement character".
A rule like this would prevent the admin\U+D800 vulnerability because a codec is not allowed to behave this way (truncating or replacing). With JSON, this isn't specified, so systems where the same data is parsed by multiple subsystems are vulnerable.
Specifically regarding the admin\U+D800 problem:
Any sufficiently complex system will have multiple subsytems.
So say you have a user-facing system that accepts a "new account" request in JSON. The attacker passes in "admin\U+D800" as the group (meaning the 5 character "admin" plus the unicode character D800, which is invalid).
Your account creator's JSON decoder ignores bad characters, so when it checks the group vs its blacklist, it finds that "admin\U+D800" != "admin". Validation passed, it sends the new user info to the account storage subsystem (as a JSON message).
Your account storage subsystem's JSON decoder silently strips bad characters, so your account subsystem only sees "admin", not "admin\U+D800".
Now this new user is an admin, thanks to a privilege escalation vulnerability brought on by a lax JSON spec (all codecs involved were conformant).
If the JSON spec had specified that "A codec MUST reject invalid characters. It MUST NOT truncate them or replace them with the Unicode replacement character", this kind of vulnerability could not happen with a conformant codec.
Thanks for the feedback!