Csexp: S-Expressions over the Network

I think a lot of data encoding protocols that are for arbitrary non-app-specific data needing more structure than Unix shell tools lines&whitespace, and intended to be simple, but you still want to make them efficient to parse while streaming, end up like this. You have a few types, some values are variable length, historically you often choose ASCII character set (for human readability, portability to different platforms, passability of different gateways), and you might as well minimize parsing lookahead, so there's some likely ways you'll signal types and sizes in the stream.

Google Protobufs (for another contemporary popular protocol) has somewhat different requirements, which leads to less-text result. But if you had those requirements (say, extensible types like in a traditional rich RPC mechanism, and maybe minimizing bandwidth), you might end up doing much the same thing.

Or, today, you and GPT-4 might just punt on "micro-optimizations", and layer atop JSON or maybe still XML.

The OP link is not part of the Racket language or its standard library. It is simply the documentation for a package written by a member of the community and hosted on Racket's official package platform — like putting something on PyPI if you're familiar with the Python ecosystem.

The package itself is listed over here [0], which links us to the source code [1], which in turn attempts to link to a URL that suggests the idea is due to Rivest [2], but that URL has apparently died.

[0] https://pkgs.racket-lang.org/package/csexp

[1] https://gitlab.com/spritely/racket-csexp

[2] https://people.csail.mit.edu/rivest/Sexp.txt

if you were sending a pure, anonymous function, that would work fine. But if you're making a closure that makes use of any surrounding variables, libraries, or state, that's a non trivial set of information to transmit across the wire.

Mutable state with serialise/deserialise either breaks the aliasing or needs some way to keep the referenced variables alive (nasty interaction with garbage collection) and to re-establish aliasing on deserialise.

Serialising a DAG to a tree tends to duplicate the previously shared nodes. There's then the question of whether deserialising the tree should re-establish the original sharing or some other sharing. Serialising a graph to a (finite) tree needs some notation to represent the cycles.

It looks to me like mutable state is the blocker but I'd be interested in other opinions.

    (lambda () y)

You can’t just print the original source definition, as y would then be undefined (or I suppose it could be defined to be whatever y is in scope at the deserialization call site or something thereabouts, but that wouldn’t guarantee that the deserialized function is equivalent to the other with respect to how it behaves, despite the source being identical).

You can’t just inline the serialization of y, because the original function returns a particular reference, and there’s no guarantee that the rest of the dependent code will not expect a stable, unique (and possibly mutable/state-full) instance of this value.

I suppose you could, on the side, also serialize the entire program state (or rather, at least the stack and heap), but then things get tricky when any object refers to something like an opaque pointer (even if you have an extent you can safely copy, you’d have to handle issues like different virtual addresses being mapped) or file descriptors, etc.

But maybe all those challenges can be overcome — so I’ll ask: how would you go about serializing an arbitrary object in such a way that one could deserialize an equivalent object?

For now, I encode everything as JSON, then decode back into lists, etc. on the other side (using cl-json). I was hoping there might be something more direct. But who knows, maybe the JSON libraries are so optimized they’re faster anyway!

Escaping strings and parsing floating point numbers isn’t something I enjoy.

I’ll look at them more closely.