Hacker News Reader

Comment by stinkbeetle | Hacker News Reader

With these serialization libraries, do any of them have a facility that allows you to specify a wire format and an application format, with recipes for converting one to the other?

I haven't used these very seriously but a problem I had a while back was that that the wire format was not what the applications wanted to use, but a good application format was to space-inefficient for wire.

As far as I could see there was not a great way to do this. You could rewrite wire<->app converter in every app, or have a converter program and now you essentially have two wire formats and need to put this extra program and data movement into workflows, or write a library and maintain bindings for all your languages.

View on HN

>> You could rewrite wire<->app converter in every app

This is what Google does. We joke that our entire jobs are "convert protobuf A into protobuf B".

The way to do this starts with not hard-wiring the code generation step.

Instead, make codegen a function of BOTH a data schema object and a code template (eg expressed in Jinja2 template language - or ZeroMQ GSL where I first saw this approach). The codegen stage is then simply the application of the template to the data schema to produce a code artifact.

The templates are written assuming the data schema is provided following a meta-schema (eg JSON Schema for a data schema in JSON). One can develop, eg per-language templates to produce serialization code or intra-language converters between serialization forms (on wire) and application friendly forms. The extra effort to develop a template for a particular target is amortized as it will work across all data schemas that adhere to a common meta-schema.

The "codegen" stage can of course be given non "code" templates to produce, eg, reference documentation about the data schema in different formats like HTML, text, nroff/man, etc.

I didn't recognize the GSL citation, so for others:

- https://github.com/zeromq/gsl/blob/v4.1.5/examples/fsm_c.gsl

whew, this readme has everything

- XML in, text out: https://github.com/zeromq/gsl#:~:text=feed%20it%20some%20dat...

- a whole section on software engineering https://github.com/zeromq/gsl#model-oriented-programming

- they support COBOL https://github.com/zeromq/gsl#cobol

- and then a project 11 years old with "we're going to document these functions one day" https://github.com/zeromq/gsl#global-functions

What a journey that was

If you care about network bandwidth you can compress before sending, as virtually all web applications do. Then you don't need to worry much about the space efficiency of the application format.

Of the wire format you mean? I compress it and still need to care about the space efficiency of the wire format beyond that. Compression ratio does improve a lot when not doing our own, end result is significantly larger. Also it becomes also significantly slower because more data to process which is possibly the bigger problem.

It's probably not like most web application, it's hardware data loggers that produce about hundreds of millions to billions of events per second (each with minimum about 4 bytes of wire format and maximum roughly 500 bytes).

Reply on news.ycombinator.com