JSON for Modern C++
github.com
github.com
In every application, any dependency to JSON should be minimized, contained and preferably eliminated.
What's wrong with writing "initializers" which are serializers/deserializers? If you're looking for automatic file format to C++ class object, why settle for JSON (whether it's this library or JSONCpp) why not use Thrift or Protocol Buffers?
- Thrift: https://thrift.apache.org/ - Protocol Buffers: https://developers.google.com/protocol-buffers/
Because I'm not writing code in a vacuum, and JSON is what everyone else is using. JSON is chosen for simplicity & interoperability.
Maybe JSON is what you get sent and you've got no control over that? Or JSON is what you want to expose and you've got no control over that either? You can want to achieve reasonable performance while still meeting external requirements.
This has carried me for 99% of all use cases along with a try/catch that barfs and an array specialization that makes a vector:
template<typename T> T get(const char* key) { return boost::lexical_cast<T>( json.GetObject(key) ); }
The reality is that if I see some value of a type I wasn't expecting, something probably went wrong or changed in the protocol anyways.
The idea that types can replace code is just broken. The type system is not a programming language. Or at least, as proven by C++/Haskell/..., not a remotely usable one.
It's obviously not simple - no two parsers have the same behaviour!
To be clear, I was not talking about lexing JSON, but about how to configure a "meta-parser" like this library to actually convert some JSON to concrete application data objects. The point of contention was, "should we really use such a massive library or mustn't it be sufficient to have a simple bag of parsing primitives (which we can use to actually build the right thing)".
And I'm not a fan of JSON either...
> Tip, if you want to defeat an argument by quoting only three words, make sure you're talking about the same thing.
Please don't be snarky - if you disagree say so and why.
My apologies. Just did that - I'm habitually a post-then-improve type of commenter.
This is the exact nonsense that is happening all over the place, and why we need 512GB ram in workstations to compile something.
I can't help but think that some basic knowledge about libraries is missing.
The JSON obsession is that everything under the moon supports it- and it's a nice easy structure which can be converted to a dynamic or statically typed object with no issues. Plus it's very human readable, so it's great for debugging.
If you sit there and ONLY use C++, ok why bother? But many systems need to talk to different systems these days. I can setup a webapi that can communicate with my angular typescript, C#, matlab all through nice easy JSON. Each language just converts it to the object and I'm good to go.
The question is why are you so against such a widely accepted format of communication?
Of course there are other data transport protocols, XML and ProtoBuf to name a couple off the top of my head, but those are extremely heavyweight.
If you want to use JSON for config files, again, they are easy to make human readable and parse. Grab an off the shelf parser and support whatever scenario users may need. Sure we have yaml and ini, or whatever key=value config format your project creates, but you'll be much more on your own with how deep you want to support various value types.
As far as your complaint or desire for JSON to be minimized or eliminated as a philosophy, I'm not sure why this is the top comment about a very well made and loved C++ library to parse JSON. Are you hoping your comment will change people's minds about what library to use or whether their system needs JSON at all? I don't really understand your goal.
Hence, our embedded C++ backend can now easily accommodate and return JSON to the front end using this library.
Plus, what data format do you propose for things like configuration files, etc? No matter what format you choose, you are going to need a parser of that format for C/C++.
TOML has inline and multiline forms for most objects.
This is my entire ini parser for making a std::map<std::string, std::string>. I cast/parse the values at the use site:
std::regex section_test("\\[(.*?)\\]");
std::regex value_test("([\\w\\.]+)\\s*=\\s*([^\\+]+(?!\\+{3}))");
while(-1 != getline((char**)&buf, &len, file))
{
std::string line(buf);
trim(line);
if(!line.size() || line[0] == '#') continue;
std::smatch match;
if(std::regex_search(line, match, section_test))
current_section = match[1].str();
else if(std::regex_search(line, match, value_test))
{
std::string key(match[1].str());
std::string value(match[2].str());
info("INIReader: %s::%s = %s\n",
current_section.c_str(), key.c_str(), value.c_str());
config[current_section][key] = value;
}
}We got used to this, but results are pretty ugly. We get people confused about delimiter order, broken copy-paste, and so on. For the next project, I’d recommend something which can support complex structures. A schema validation would be nice, too.
I put this code in a standalone cpp file so that it compiles to its own object in parallel with the rest of the system. I've never had issues with it with regards to build times.
how do you suggest interaction with $standardised_protocol which uses JSON in that case ?
And how to make the transformation? Using your own setup. No point in expressing (duplicating!) your own data structure definitions in a random library's DDL (which, apart from the inflicted duplication, will not fit very well since it doesn't know your project).
I don't understand, doing this step is exactly why you need to have a library such as the one which is being discussed here. There's no in-language way in C++ to go from a string that looks like `{ "foo": [1,2,3] }` to `struct { int foo[3]; }`.
I would just specify your case as a data item:
ARRAY_ITEM("foo", foo, INT, 3)
The corresponding code that handles the data items of kind ARRAY_ITEM would be something like this:
case TYPE_ARRAY: {
char *store = add_offset(&myStruct, spec->offset);
int elemSize = typeKindToElementSize[spec->typekind];
JsonArray *jsonArray = get_json_array(jsonObject, spec->srcfield);
for (int i = 0; i < spec->arraylength; i++) {
JsonValue *jsonValue = get_json_array_element(jsonArray, i);
/* This makes sure the dynamic type of jsonValue matches, then
does the appropriate conversion and stores the thing. We also
want to check errors, either now or at the end. */
store_json_value(jsonValue, spec->basetype, store + (i * elemSize));
}
}
I'm likely to write a new version of this for every new project, to avoid dependencies and to allow for change.
struct MyWidget {
void setFoos(const std::vector<int>& foos) { m_foos = foos; updateUI(); }
};
struct MyWidget {
void addFooInstance(int);
};
As they say : thanks but no thanks, I'll stay with `for (auto& [key, value] : o.items())` which ensures that everyone including the fresh-out-of-school student can understand what happens without needing to read obtuse macro definitions
I say let data be data and write code where you need code. How does the library handle this in one step? How long does it take you to find out?
> Also, ugh. macros.
Data macros are the best. You don't have to debug them at runtime, they let you get rid of a lot of boilerplate such as compile-time computations (e.g. offsetof(), sizeof())
> do you now have ARRAY_ITEM_JSON, ARRAY_ITEM_BSON, ARRAY_ITEM_CBOR, ARRAY_ITEM_YAML
I'm really sorry if you have to do that. I never had, and likely never will. But I'd probably just write the example I gave, in 5 flavours, in 5 separate implementation files. The alternative, dealing with 5 oversized libraries that approach this thing in a totally different way, is not appealing to me at all.
> What when the format changes so that `ARRAY_ITEM("foo", foo, INT, 3)` now wants floats ? woohoo, magic truncation instead of a compile error.
No, runtime error message about a wrong type in a JSON payload. Or if you mean this: the type of the C array's elements changed from int to float. Then just check the type of the array elements against the specified type (INT) which should expect a specific C type. There are easy ways to get a representation of the array element type in C++ as well as an in C (the macro can do it automatically).
> obtuse macro definitions
#define ARRAY_ITEM(_srcfield, _arraylength, _basetype, _dstfield) { .srcfield=_srcfield, .typekind=TYPE_ARRAY, .arraylength=_arraylength, .basetype=_basetype, .offset=offsetof(MyStruct, _dstfield) }
If you think that is obtuse reconsider C++ templates.
Since JSON only has basic data types, it's easy to just parse and retrieve objects, similar to using XML tree parsing libraries back in the day.
Of the myriads of existing transfer or serialization formats,
* JSON is still the only secure by default one (if you ignore the two later updates, which made it insecure),
* JSON is by far the easiest to parse (small and secure, no references),
* is natively supported by Javascript.
It has some minor design mistakes, and is not binary (such as msgpack, which is therefore faster), but still is a sane and proper default. Esp. it does not support objects and other extensions, with its initializer problems on MITM attacks, and is properly ended. Unlike XML, YAML, BJSON, BSON, ... which do have major problems, or msgpack, protobuf, ... which do have minor problems.
In every application, any dependency to XML should be minimized, contained and preferably eliminated.
And enough people put it into eval that several companies started prepending while(1); to their JSON messages. Don't blindly trust user input no matter what format it comes in.
> JSON is by far the easiest to parse (small and secure, no references),
We all know its definition fits on a business card, which is the reason we have at least seven different specifications[1] on the details omitted from the card.
while(1); existed to work around a browser vulnerability where the same-origin policy could be bypassed by including cross-origin JSON as a <script>. Nothing to do with eval.
How can a data format be insecure?
> JSON is by far the easiest to parse (small and secure, no references),
It's relatively easy to parse (while not as easy as a sane well-specified version of CSV would be). But it also offers no structural integrity, which means that you need to augment the parsing code with terribly ugly validation code. (I don't really use JSON frequently, but the attempts at providing a principled validation framework on top of JSON that I've seen were... unsatisfying).
> In every application, any dependency to XML should be minimized, contained and preferably eliminated.
That's missing the point. It's not about JSON vs XML vs whatever. It's that application logic shouldn't be operating on transport data formats. And even if JSON has a simple representation as runtime data objects (in most scripting languages), that representation is pretty far from ideal compared to a representation tailored towards your specific application logic.
If it tries to be too featureful, then it requires parsers to be very flexible. Malicious input can trick a too-flexible parser into doing things that the developer of whatever function called the parser probably didn't want to happen.
For example, YAML:
https://metacpan.org/pod/Cpanel::JSON::XS#SECURITY-CONSIDERA...
which just misses details on stack-overflows on overlarge nesting levels, or denial of service attacks on overlarge strings, arrays or maps.
Better formats which prepended sizes do have an advantage here, such as msgpack. But msgpack has no CRC or digest verification to detect missing or cut-off tails. JSON (its secure 1st RFC 4627) must be properly nested, it does not need this. From the 2nd RFC 7159 on it became insecure, and the 3rd RFC 8259 is merely a joke, as it didn't fix the known issues, only removed a harmless feature.
Since UTF-8 is the only sensible format for JSON it makes little sense to require UTF-16 and UTF-32 support. ( unless you have some special requirements on encoding, in which case you can just disregard that part and convert it on both ends yourself )
The only "issue" with non-object values I see is the one mentioned in the above link where naively concatenating JSON might lead to errors when you send two consecutive numbers but that's going to rarely happen so your system can just reject top-level numbers if it doesn't expect them. And even then the simple solution is to just add whitespace around it.
schmorp wrote this:
> For example, imagine you have two banks communicating, and on one side, the JSON coder gets upgraded. Two messages, such as 10 and 1000 might then be confused to mean 101000, something that couldn't happen in the original JSON, because neither of these messages would be valid JSON.
> If one side accepts these messages, then an upgrade in the coder on either side could result in this becoming exploitable.
The 3rd version was a joke, because the outstanding problems were not addressed at all, and removing BOM support for the 4 other encodings is just a joke. First, you cannot remove a feature once you explicitly allowed it, esp. since it's a minor and an almost unused one.
And remember that http://seriot.ch/parsing_json.php was already published then, and the most egregious spec omissions had been known for years already. such as undefined order of keys, or undefined if duplicate keys are allowed. Allowing unsorted keys is also a minor security risk, as it exposes the internal hash order, which can lead to hash seed calculation.
Wrong. CSV is horrible to parse with its string quoting rules. JSON accepts only utf8 and is not misleaded by \, nor ",". Done both, JSON is much simpler and can represent nested structures, arrays or maps. CSV importers are mostly broken, and should be left to Excel folks only.
> It's that application logic shouldn't be operating on transport data formats
Nobody is talking about application logic here but you. JSON is one of the best transport data formats, and this library makes it much easier to encode/decode from/to JSON and C++.
We are not talking about javascript's hack to prefer JSON over full objects. Of course is an internal representation always better than an external, esp. such a simplified one. But for transports simple ones are preferred over complicated serialized ones, because then you can easily do MITM stack-overflows or abusing side effects on creating arbitrary objects.
If you didn't do a compilers class and you want a simple language to play around with for lexing/parsing, JSON works great. Here's the core of a probably correct JSON lexer, albeit a super inefficient one, that I whipped up in <110 lines of Python a few years ago out of curiosity[1].
By comparison, check out the state machine transitions at the heart of the cpython implementation of the csv module[2]. It's not really a fair comparison (my JSON lexer is written in Python and uses regexes, the csv parser is written in C and does not use regexes) but even ignoring how nicely the csv parser handles different csv dialects, I still find it strictly more complex.
[1]: https://github.com/chucksmash/jsonish/blob/master/jsonish/to...
[2]: https://github.com/python/cpython/blob/41c57b335330ff48af098...
All I'm saying is that flat database tuples are even easier to parse than JSON (which is nested, so requires a runtime stack). It was a total side note (in parentheses!).
My main argument is that JSON is a mess to validate.
> Wrong. CSV is horrible to parse with its string quoting rules.
I made a reasonable attempt at preventing exactly this misunderstanding, but I guess if people want to misread you they will.
> Nobody is talking about application logic here but you.
The application logic (I'm including internal data representation here) is the meat of all efforts, so you should absolutely be considering it, instead of needlessly putting isolated efforts in optimization of side actors. Parsing JSON should IMHO be a total side concern, and using an oversized library like the one we're talking about fails to acknowledge that. Such a library is either a wasted effort (if its data types aren't used throughout the application), or else (if they are) use of the library leads to massive problems down the road.
Note that the quote is talking about a well-specified CSV, not any CSV in general. A well-specified CSV would indeed be fairly easy to parse.
RFC-4180
Yes of course there was no recognised standard before that. Just like before Greenwich Meantime there was no recognised standard for universal time coordination ...
The situation in practice is that when people want a “standard” way to do CSV, there is in fact a standard they can use, that does cover most sensible things you’ll want to do with CSV, and addresses the most common corner cases (eg delimeter in field) in a fairly sensible way.
You are free yet to make whatever proprietary extensions or otherwise, at the risk of losing compatibility just as you are with any other standard.
XML external entities allow for arbitrary file inclusion: https://en.wikipedia.org/wiki/XML_external_entity_attack
You can make a badly configured XML parser allocate memory until it crashes: https://en.wikipedia.org/wiki/Billion_laughs_attack
You cannot host user-generated xmls on a domain without making yourself vulnerable to cross site scripting attacks. Browsers will happily execute any javascript you include: https://stackoverflow.com/questions/384639/how-to-include-ja...
Often you have control over the sending and receiving parts of an API and can design the transport format to fit your application logic.
For example, I am currently using a GraphQL API and the data structures I get from this API are exactly what I need in my application.
Of course, it's not limited to JSON. For applications, it should be a tiny insignificant detail what format is used for serialization. Be it JSON or some superior format, such as XML.
It depends on the extant to which your program needs to communicate/interoperate with other programs...in particular ones without the same programming language (no Java RMI).
Under a "Unix" philosophy, this happens quite a bit.
Links:
http://rapidjson.org/ https://github.com/Tencent/rapidjson
https://github.com/open-source-parsers/jsoncpp
https://www.codeproject.com/Articles/20027/JSON-Spirit-A-C-J...
Our primary use is to parse http://cocodataset.org/ metadata files and RapidJson is 100% faster.
So, twice as fast - it parses the dataset in half the time?
Previously we used rapid json which is faster but the syntax of the nlohmann json is nicer and it is much quicker to develop with.. we also use python a lot so the familiarity between the 2 is also useful, some of the other C++ json libs can have quite convoluted api’s for dealing with values, objects and arrays.
The overwhelming majority of our IO is reading / writing data files, but those are stored in an optimized binary format. The configuration / metadata accounts for a much smaller fraction of IO. For this tiny fraction we care about flexibility and readability, a slow JSON parser is fine.
At some point it seems like a general mindset shifted from making things efficient at every level to assuming things don’t matter if you’re probably doing something worse anyway.
If reading/writing is the hot path things are different, though.
I would point out that its convenience in large part depends on extensive use of cast operators, which can lead to some tricky corner cases. It is also the only component in one of my projects that confuses xlclang++ on AIX.
If I had enormous globs of JSON to read I might use something faster (I believe there's a very fast library that uses SIMD instructions) but my needs are quite limited: basically exchanging snippets of JSON with a network service. So clarity was far more important than performance.
Writing documentation, and easing users into your project, is an underrated skill.
I personally prefer a doc/ folder with RST files in it. That's what the Linux kernel does.
A doc directory gets you the same thing, easier.
(It is actually possible to use without, but then exceptions become aborts, and you have to dance around that.)
Is that because the python dictionary happens to looks so much like json, or what do they mean?
Maintenance note: This repo is stable but no longer actively maintained. No further development is planned, and no new feature PRs will be merged. Bug fixes may be merged on a volunteer basis.
With JSON being an unchanging spec this makes sense.
It's basic and has no compilation speed issues.