Sharing data types on a multi-language project | Hacker News Reader

Sharing data types on a multi-language project

mortoray.com

19 pointsingve9 comments

> Protobuf is another library to handle cross-language serialization, but wouldn’t offer us the flexibility to add all the other features we now have — plus, it was a binary serialization when I last used it.

Protobuf has a well-defined text format, which is the top search result for "protobuf text" [1]. Proto3 also has a clearly defined set of rules for supporting JSON.

[1] https://protobuf.dev/reference/protobuf/textformat-spec/

CORBA before that, solved the same problem. More recently l, when in automotive, I worked with Franca IDL [1] a lot, which contains a more modern take on this. Seems this issue is solved over and over again, but for some reason so far no solution stuck.

[1] https://en.m.wikipedia.org/wiki/Franca_IDL

ACE TAO didn't, but it was a long time ago and I was probably holding it wrong.

This is a very intriguing problem!

Not trying to steal thunder, but quicktype (https://github.com/quicktype/quicktype) does all of this–JSON schema or TypeScript definitions to de/serializers and types in nearly 30 languages at this point. Try it here: https://app.quicktype.io

Why would JSON Schema not work here? Both languages have libraries to emit/parse.

Either that or OpenAPI Specification (which is in some ways a superset of JSON Schema). At a previous job, we had OpenAPI specs that were mostly just schemas with a few endpoints defined. The tools around generating OpenAPI clients are robust and mature.

Taking this concept further one could model in RAML [0] to define both the types (flat or nested) and api definitions. It's based on YAML 1.2 with enough maturity to provide capabilities such as union types, extensions, includes, user-defined facets, etc.

The AMF project [1] can be used to parse and transform to/from RAML, OpenAPI, GraphQL, and json schema. Code generation to languages of choice can be bolted on from there.

I'm using this approach to define canonical data models. Subsequent code generation scaffolds internal application integration apis, master data management (MDM) entities, and SQL/OLAP artifacts for ETL / BI purposes.

This approach keeps overall end-to-end data architecture consistent, in sync, and versioned under source control. Additionally, flat types as required by relational systems are re-used and composed into nested complex types more appropriate for apis. Metadata is layered on as needed to refine the models for system-specific needs, for example to add user-facing field groups, descriptions, and formats for BI datasets, sensitivity levels and other data security controls, business rule definitions for MDM, etc.

[0] https://github.com/raml-org/raml-spec/blob/master/versions/r... [1] https://a.ml/docs/

From glancing over the article, I am not sure how type_spec is actually invoked. Is it a separate build step that emits the language-specific type-definitions?

Pardon an old curmudgeon: I'm sure this improves on SWIG in myriad ways, but how, exactly?

Reply on news.ycombinator.com