Hacker News Reader

> For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having to sacrifice type safety to get there.

This has not been remotely my experience. Deserializing has a lot of implications for ownership and you get into complicated deserializer trait specifications pretty quickly.

In particular, there doesn’t seem to be a way to deserialize a type that has a reference because you don’t seem to be able to tell serde who should own that memory. Maybe I’m wrong, but I’ve walked through this multiple times with experienced Rust users and no one could get it working.

Serde is impressive, I’m sure, but far from “don’t even have to think about it”.

We almost always deserialize to owned data, partly because our deserialization source is almost always compressed so the data has to be copied. There are a few cases where we want to serialize with a reference and deserialize to owned, but that's no big deal using custom "with" functions.

We have hit one serde footgun, which is that skip-serializing-if corrupts your data with formats like bincode :-(. https://github.com/serde-rs/serde/issues/1732 really wish that could be fixed.

> Is there any actual reason something like serde couldn't exist in other languages?

C# standard library has DataContractSerializer for decades. That thing reads/write text XML by default, but with minor tweaks can do binary XML or JSON.

I like their binary XML format the most. Very fast because doesn’t waste time printing or parsing numbers or Base64 bytes. Also, pre-shared XML dictionary, and session-accumulating dynamic dictionary, makes the serialized representation very compact, sometimes an order of magnitude better than text/xml.

These sorts of things are really convenient, but the idea of tightly binding your type definitions to serialization formats always makes be a little nervous: I haven’t used Rust enough in practice to have an opinion here, but I’m wary of a solution in this space that makes it too easy to ignore the forward/backwards compatibility of your external format.

Yeah, for sure that is something to be cautious of. For cases where I expect backwards compatibility to be a concern, I will usually make the top-level type of the message or format an enum with each variant being a version (initially with just a single version, V1 or V0). That way, if we need to make backwards-incompatible format changes, we can create a new variant, and potentially even automatically migrate between formats (I've done this a few times). Then all we have to do is call a function to migrate to the latest format, and we can use whatever that is for all the actual program code (generally it will be a type alias for the version structure of the same name). Of course, if this gets too complex or you start to have non-treelike relationships between values, you still want to fall back on a more traditional database schema, but this kind of versioning can get you a long way.

That said, it's incredibly refreshing to be able to focus on these semantic and migration concerns without having to worry about how to deal with parsing them later. serde doesn't solve enforcing backwards compatibility for you, but it sure makes doing the right thing a lot easier. You never have to worry about some code accidentally forgetting to check the version field, people carelessly mixing and matching messages with different versions, or fields that "should" never be set but are there for backwards compatibility. That's worth a lot to me!

I rarely have to think about deserialisation in c# or java either...

I wrote a lot of Java back in the day. I elaborated on why I disagree in the other comment. I do not know if the situation in C# is similar.

For context, the example from the Serde wiki written in Java using Jackson

  public record Point(int x, int y) {}

  // [...]

  var objectMapper = new ObjectMapper();
  var jsonString = objectMapper.writeValueAsString(new Point(2,1));

It isn't obvious to me what advantage Serde offers that you couldn't get with Jackson or other similar libraries in Java (although I get that Java and Rust are different languages, and that there may not be something this ergonomic in the likes of C++)

For that example there's no advantage. But it's really, really easy to find examples where there are (the simplest example is, anything with pointers with @NonNull annotations, or anything which is performance-sensitive and contains other objects).

That has been the same in Java and .NET world for 20 years now, just plug the desired serializer from the standard library and be done with it.

It absolutely does not provide the same guarantees, and I've covered why in several posts now. Do you believe that people who disagree with you have never used any language other than Rust before? Having to deal with data structure invariants getting violated due to reflection is a huge headache and makes the parsers way less useful than they would be otherwise.

Yes, because those same people seem to be unaware reflection is not the only way to use such APIs, revealing a superfical knowledge of how those ecosystems have been working for the last 25 years.

How is it implemented in Java or C# if not via runtime reflection? I've always assumed that's the way its done...

Java has annotation processors which can perform code generation, just like Serde does.

Manifold [1] an example framework that relies heavily on that, achieving similar things like Serde.

Java developers don't like to use these things too much though, unlike Rust developers who love their `#derive`, so you're mostly right that reflection-based serialization is still more common, but that's by choice.

Dependency Injection is in a similar situation: frameworks like Micronaut [2] can do it without reflection, and things like Google Dagger [3] have existed for several years that do the same thing... but still, most Spring Boot-based projects I know of use reflection-based DI as well... it's fast and the security issues have been largely mitigated nowadays.

[1] http://manifold.systems/

[2] https://docs.micronaut.io/latest/guide/

[3] https://rskupnik.github.io/dependency-injection-in-pet-proje...

Thank you for the detailed write up! I love finding an out about cool language feature/tools

I'd always recommend folk use constructor-based deserialisation, rather than getting the deserialiser to create empty objects and poke at fields.

That's still (at least naively) using reflection, but it's a lot safer.

In practice, using reflection for every object being deserialised is far too slow and Jackson at least generates code at runtime rather than repeatedly reflecting. That's magic, but it's nice magic because it's not breaking the language: one can see that it's possible to implement it in pure Java.

I don't program Java, but I have implemented and used similar things for C#.

Reflection is indeed used there, but not to serialize. It's only used once per type, to generate code.

C# has multiple ways to generate code in runtime. One is Reflection.Emit, allows to manually generate bytecode instructions + metadata. For instance, one can build new types in runtime. A typical pattern is implementing manually-written abstract class or interface with generated types: this way the manually-written code can call into runtime generated one.

Another method is System.Linq.Expressions. This one allows to generate code (no new types though, just functions) from expression trees, and provides API to build and transform these expression trees.

Regardless on the method, the generated code is no different from manually written code. JIT compiler can even inline things across, when generated code calls manually written one, or vice versa.

Ah the bytecode generation via Reflection.Emit seems like a really powerful feature. Does it feel "brittle" to use, or is it expressive enough to aid writing more robust code (to a degree, I realise it's not possible to aliminate every problem)?

There's a weird (not in a bad way) elegance here that reminds me of lisp.

> Does it feel "brittle" to use

Yes and no.

No because when you try to do unsupported things like calling a method on an object which doesn’t support one, you gonna get an appropriate runtime exception.

Yes because if you fail lower-level things like local parameter allocation, you gonna get an appropriate runtime exception but that one is (1) too late, I’d prefer such things to be detected when you emit the code, not when trying to use the generated code (2) Lacks the context.

Overall, when I can I’m using that higher-level System.Linq.Expressions for runtime codegen. Things are much nicer at that level. I only using the low-level thing when I need to emit new types, like there: https://github.com/Const-me/ComLightInterop/blob/master/ComL...

You can use stuff like VS T4 templates or compiler plugins via annotation processors.

So at compile time, just like serde works, you create your type safe serializer.

Why would reflection break invariants? Most deserializers can pass through data into the constructor via reflection as well, where you can protect against invalid objects.

Yay, except that at least Java does so by using reflections and has lead to more then just a few massive security vulnerabilities.

Just to be clear, there are Java libraries which do thinks better in a less problematic way, just the facilities shipped in the standard library are IMHO inadequate.

Macros are definitely key. I wonder if ownership helps too? Circular references are always weird when serializing