Hacker News Reader

> What about serialization makes it so intractable? Could some kind of language support make it better?

I've thought about serialization a lot over the years, and I think a big part of the problem is that we try to map our programming language data types directly to serialized types (e.g. mapping JSON objects to Rust structs). That can be useful, but I think often the serialized representation is sufficiently different from the desired data type that it's not good enough. So we end up with one set of data types for serialization/deserialization, and a bunch of code that maps between those and the desired representation.

I wonder if a system similar to Scheme's `syntax-rules` (or Rust's `macro_rules!`) for mapping the serialized data into the program's data types (and vice-versa) would be a nice mechanism.

> I think a big part of the problem is that we try to map our programming language data types directly to serialized types

Much of the time, in-program data is relational with a possibly-cyclic graph describing relationships, ownership, etc. Systems that work with RDBMSes do a fine job of de/serializing data for relational stores, and ORMs are a powerful and delightful way to do this.

The problem with JSON and XML for these kinds of data is these formats are inherently non-relational. They are simple trees of data, not arbitrary graphs. In practice, e.g. in every full-stack web codebase I've ever seen, JSON blobs are hacked together e.g. by doing a makeshift front-end "join by ID" when needed — or adjusting a back-end method to perform the join and return — wait for it — a flattened JSON blob.

It seems like we need a "JSON" or "XML" where relationships are a first-class citizen. To my knowledge, the closest we've come to this is when the data broker and format are tightly coupled, e.g. with SOAP or GraphQL. Can this be achieved strictly declaratively, in a format that's better than a SQL data dump? Maybe!

> I wonder if a system similar to Scheme's `syntax-rules` (or Rust's `macro_rules!`) for mapping the serialized data into the program's data types (and vice-versa) would be a nice mechanism.

Does `serde`[1] look like it's close to your imagined mark? I haven't tried it personally, but it looks akin to Java's `Serializable` or C#'s `ISerializable`— all reasonably elegant ways of handling the messy binding logic of serialization over arbitrary formats.

[1] https://serde.rs/

> and ORMs are a powerful and delightful way to do this

I think we'll end up disagreeing there.

However, I think this type of model could maybe work better for "static" serialization/deserialization of file formats. But even so, I think the biggest part that's missing is a convenient way to describe the mapping between the structure of the serialized data and the internal representation. ORMs typically give you a single method of doing that mapping, which can work when you have control over the serialized representation. But it doesn't work very well when you're serializing to an existing file format, for example.

> Does `serde`[1] look like it's close to your imagined mark?

No, serde is exactly what I was talking about in terms of the serialized data matching your internal data structures. I think it's a necessary piece, but I think the part that's missing is a nice way to map that data to the representation your program want's to work on.

> I want rich applications that work in all languages (including input methods) and also play nicely with accessibility tools such as screen readers. That can't be, and shouldn't be, a tiny little executable.

Apple historically did a pretty good job of solving this by including a rich GUI framework in the OS. Even now, from what I've heard, that approach seems to work pretty well for Mac-only apps. But then, in your xi retrospective, you said the "native" GUI wasn't all that great even on Mac. And now, except for the indie Mac crowd, developers are generally expected to target multiple platforms. So now, I guess applications have to carry with them something that was formerly expected to be part of the OS.

> What about serialization makes it so intractable? Could some kind of language support make it better?

I think the main problem with serialization is that it is used to mean several different things which at first glance could all be handled by one system but experience instead shows are just too different to be handled in that manner.

One kind of serialization is handling something that is mostly "bit blitting." I've got a struct in memory, it needs to be copied to disk to be read back into memory. Maybe you need to do some normalization for things like endianness or charset conversion.

A different kind of serialization is when you're trying to read/write a standard-specified binary layout. This sounds like it's the same thing as before, but the difference here is that the structs are not readily blittable. You might have cases like x86's definitions, where fields are splayed out in different locations (zmm0 is spread out across three locations in the XSAVE struct). Or you might have the ... interesting features of ASN.1's DER format. Handling these kinds of cases requires at the same time as a more blittable struct requires adding lots of customization points.

A third situation is where you're reading in something like JSON or XML: now you have symbolic field names that can occur in any order, and your serialization code needs to account for unexpected orders. Customization points really kick in here, because now you also usually want to have default values and optional values and maybe conditionally required (this field is required only if some other field has a particular value).

On top of these, you also start to conflate orthogonal issues of validation and conversion. Are the structs in memory meant to be close to on-disk representations and clients responsible for handling data representation issues (consider the way the object crate handles endianness in ELF files)? Where, and when, does validation of parsing data happen? How much of this is automatable from the serialization framework itself (e.g., the value must have values in such-and-such range)? What about supporting multiple versions of save data--where is the ground truth about how to upconvert (or maybe even downconvert!) old versions to the current library version?

This is a situation where you can build something pretty tight for a single use case, but feature scope quickly comes in when it comes to extending it for multiple use cases, to the point that it's too unwieldy if you're not using any advanced features and the advanced features are too advanced to use.

As for what language support makes it better, the obvious answer is reflection. Having written my own auto-derive-serialization crate, what I mostly need is a) given this struct, give me a list of fields, their names, types, and attributes, b) let me do specialization on types, and c) let me turn that into writing some pretty unconstrained code. Rust's current macro system mostly falls down on the type system, although the need to include a hefty parsing quote to reparse everything to get the field lists is pretty poor (not to mention actually handling the corner cases in Rust's type system is challenging, although serialization code is likely to screw up here anyways). There's probably room to specify a MVP to make implementing serialization easier that wouldn't need to go through syn.

This is a great place to explore a recent line of thought I have had. I'd love to hear your thoughts on sqlite as a file format. Just hypothetically, if Runebender ditched xml and used sqlite files instead, what would that do to serialization? What other bloat implications would it have? What value would you lose by going away from xml?

It's something we thought about, but I think it's somewhat orthogonal to the question of serialization - you still need code to go back and forth between the representation in the database and the native data structures.

See https://github.com/simoncozens/babelfont/issues/10 for a deeper discussion of issues around a file format for a font editor. One of the biggest current pain points is dealing with merges in version control (usually git). I think there are ways of dealing with that in an sqlite-based file format, but I worry it would require some custom tooling, especially to make sure invariants aren't violated. Another interesting example is LibrePCB (see https://www.youtube.com/watch?v=vu-h5y6tK34) who devised their own text based format specifically to make version control more friendly.

Focusing strictly on compile time and executable size, I'm curious whether sqlite-as-a-file-format would help or hurt. Of course you still have to move from file to native structures. I just wonder how all the overhead of sqlite compares to whatever xml libs you are using. It seems that both xml and sql tables could be modeled very closely to your native structure. So the question is which approach compiles more efficiently.

Alternative text base format was very common in the past because the standard binary format used to be machine dependent for speed (almost a direct dump of the structures in memory).

Right. Dumping internal memory representations is one point in the spectrum, with extremely appealing properties for compile time, code size, and run time (basically no cost), but other serious problems, including portability (32 bits or 64 bits for pointers) and security. Many of the zero-copy serialization formats (Flatbuffers, Cap'n Proto, FIDL) are inspired by this and to some extent have the property that a "plain ol' data" struct might be serialized by a memcpy of the C representation, but try to improve in the other dimensions.

That said, I don't think dealing with the byte representation is the real problem. Projects like simdjson show that you can convert in and out of JSON very fast. The challenge to bloat specifically is getting those converted to the data types of the application. Many serialization approaches, including serde, generate a goodly amount of code for each data type, and this adds up.

Did you ever take a look at miniserde? It's pretty much just an experiment (e.g. no support for enums), but it seems like the general approach it takes would cut down the amount of code per type.

[0] https://github.com/dtolnay/miniserde

Yes. I think the characterization in TFA is still valid.

Oh, doh. It's been a while since I read this and forgot you mentioned it there.

Could you wrap the crate mentioned into wasm like watt does for proc macros?

Obviously not ideal...

Minor, but rand was mentioned in the original post and it has improved a lot.

>Now, something like half the compile time and executable size for Runebender is serialization for the XML-based UFO file format for representing fonts.

Using which library? xml-rs? quick-xml? serde + quick-xml?

A large offender is the fontinfo[1] struct, which is serialized with serde and quick-xml. Other parts (the glyphs etc) are done with quick-xml and handwritten code.

[1]: https://github.com/linebender/norad/blob/master/src/fontinfo...