Why isn't there a Swagger/OpenAPI for binary formats? | Hacker News Reader

Why isn't there a Swagger/OpenAPI for binary formats?

anachronauts.club

101 pointswoodrowbarlow50 comments

> both of these tools are only interested in allowing the author to model the logical representation of the data, not the physical. they are useful if you have data and want to automatically convert that into a file format or protocol, but not useful if you want to describe an existing file format or protocol.

OpenAPI isn't necessarily great at this either. If your JSON API wasn't designed with OpenAPI in mind (or in a way that's similar to typical OpenAPI designs), it might not be possible to write an OpenAPI spec that describes the format well. But "type: object" is valid, so it's always possible to write a "spec".

I think the question ("why is there no schema language for binary formats?") is valid, but that OpenAPI isn't as encompassing as it's laid out to be.

Yes and that's a fundamentally different problem; I think this is because there is always some kind of state in a protocol. SOAP has this problem, and also YANG[1] and tools that try to replace SNMP. You can never abstract away the state that the consumer and producer of the protocol has. There are lots of tools to disect, debug, resend and parse network protocols like IP and TCP but in those protocols we know there is a state built in that can not be broken, so we do not build OpenApi for them.

OpenApi specifications that need three or four requests to reason with are a lot harder to work with than ones that are only one request, and these are different beast they become protocols. In my personal experience the binary specification often are protocols with a lot more implicit state in them, e.g. binary diff of in memory struct etc. You quickly lose the usefullness of Swagger the more state you have.

That said there are lots of good example of binary protocols that try to be stateless.

Conclussion: This blog post need to be scoped a bit better, what kind of binary communications do you want an OpenApi for and how is that different from what exists today?

As soon as you have to encode the state of the application in the specification of the protocol (like TCP above) things get really complicated.

[1] https://en.wikipedia.org/wiki/YANG

I think OpenAPI is a rather arbitrary and approximate description of models built upon HTTP with JSON, but that's constrained enough, and the problems solved are often similar enough, for such a solution to be useful. Actually maybe JSON schema would be more useful to consider here, since that is about formats, rather than interfaces or protocols.

With binary formats, on the other hand, there's neither data model nor usage constraints like that, and a generic tool to handle all the things would be just the Turing machine -- and then we have a bunch of languages for that. As for kaitai struct and similar attempts to handle arbitrary binary formats, I hear of those from time to time, and in some cases they seem to be useful, but they too should be either Turing-complete (and then it probably makes more sense to use general-purpose languages) or would fail to be useful occasionally.

This seems obviously to be the right answer to me.

Think about handling a ZIP file: you can't start at the beginning of the file and work forward from there. No; you have to start at/near the end of file and scan for an end-of-central-directory record based on a signature (it's not at a fixed offset due to a variable-length comment field). That then enables you to find the actual offset for the central directory itself; which in turns allow you to find the files themselves.

(author here) aye, this is a crucial point. any description language for binary formats quickly approaches turing complete, and i think this is what leads to so many of the "transpile-first parser languages". i'm impressed how well kaitai has managed to wrangle that tendency towards turing-completeness; they do have an embedded expression language for attribute binding, and they do have 'repeat' and 'switch' constructs, but the language is astoundingly declarative.

i think something very similar to kaitai could work if the tools to work with the files programmatically were ergonomic and available in many different languages. e.g., if the kaitai compiler were a native compiled library with lots of packaged FFI wrappers for many languages.

Well, nothing stops Turing-complete languages from being declarative: lambda calculus is both, for instance, as well as some of the languages based on it (in particular parser combinators in Haskell and similar languages are neat and flexible).

And there's indeed a practice of embedding some languages into others, though I think it ends up quite awkward to use, and in this case potential limitations (actually I don't know if kaitai is Turing-complete) may be undesirable. I imagine in many cases it may be simpler to ship, say, a parser/decoder (and/or printer/encoder) library providing a C API that is easy to call via FFI, rather than a description in a less common language, to be used with a particular library. But then again, probably that'd have its uses too.

GRPC has a reflection protocol [1] that can be used to build Swagger style tools [2]

[1] https://github.com/grpc/grpc/blob/master/doc/server-reflecti...

[2] https://github.com/grpc-swagger/grpc-swagger

(author here) i did not know about the reflection protocol. maybe this could be a model for how kaitai struct could avoid the "centralized bottleneck" they have now in terms of ecosystem growth.

You can see that reflection in action through tools like grpcui:

https://github.com/fullstorydev/grpcui

I don't know OpenAPI and only did a quick read. Does it support any formats other than JSON and XML? If not, then it hardly seems like an appropriate analog to what you're asking for.

Can it support s-expr based formats? Plain text keyword:value pairs? Any arbitrary input? Because that's what you're asking for for "binary formats". Which is a wide-open field. To have a universal system for that you will need something closer to the parser description formats that you have already rejected. Just like you'd have to do to properly support any arbitrary plain text format instead of just JSON and XML.

OP might be conflating OpenAPI [1] which specifies HTTP endpoints of APIs with JSON Schema [2] which specifies data schemas of JSON documents. OpenAPI specs often use the application/json media type with schemas specified using JSON Schema, but they can also use other media types including binary formats. What OP is really asking is: Why isn’t there {JSON Schema,XML Schema,.proto files} for arbitrary binary formats?

Kaitai [3] mentioned by OP and sibling comments looks interesting, don’t think I’ve seen something like it before.

[1]: https://swagger.io/specification/

[2]: https://json-schema.org/

[3]: https://kaitai.io/

Automatic generation of wireshark dissectors would be amazing.

I wrote a few in Lua, but it still felt second class citizen to built-in C dissectors, loading them was annoying and the UI to use them was too.

A good dissector will annotate fields with semantic meaning and group and nest them, giving wireshark the precise bit offset+length where a value came from.

Those are some different requirements than other parsers and reuse is therefore a bit difficult.

preach! by the way: wireshark can load ASN.1 source files directly. https://www.wireshark.org/docs/wsdg_html_chunked/CreatingAsn...

Would something like this help? https://github.com/joushx/kaitai-to-wireshark

Happened to just see this article when reading Kaitai docs: https://www.incibe-cert.es/en/blog/understanding-industrial-...

I was checking out Kaitai the other day (generate binary parsers from declarative structures) and noticed its tutorials section has great content on reverse-engineering binary formats with it: https://doc.kaitai.io. Might be interesting to anyone interested in OP's question.

I like the first blog link that reverses the binary format of one of those interactive anime novels from an .exe: https://hackernoon.com/reverse-engineering-visual-novels-101...

Really cool walkthrough of how to do such a thing regardless of whether you use Kaitai to do it.

My project Restruct[1] does Kaitai-like things but also supports serialization. Unfortunately, it only supports Go and only deals with Go struct tags rather than YAML manifests. Still, it totally can be used for serialization. I use it to sketch out quick projects against arbitrary binary formats. Two examples: one, parsing PNG headers to implement a quick binwalk-like program for just PNG that looks for the IEND chunk to extract accurately[2], two, a program that splits FL Studio FLP projects by playlist track[3].

I feel like I’ve self-promoted Restruct like four times on Hacker News, and I feel kind of bad because it could use improvements and even some bug fixes and I never seem to get around to it. Oh well. It’s still useful for me, I hope it’s useful for others, too.

That said, Kaitai has a fairly clear path towards adding serialization from a design PoV; many things that would be calculated for parsing structures in deserialization could just become checks/assertions in serialization. As an example, checking that an expression calculates out to the expected value would be a reasonable approach. Reversible expressions could be implemented for some cases, too, if you want it to do more of the heavy lifting. I think the biggest obstacle is actually implementing it, and frankly my Scala is too weak to help with such a relatively big undertaking.

I’ve also played with the rust nom library, which implements functional programming style parser combinators. It is quite cool how it can express fairly complex grammars and binary formats pretty much equally well, albeit optimizing it effectively requires serious magic that I do not think nom has. (I assume in Haskell, the same thing can be done with mind-boggling optimization power.) It also doesn’t handle serialization, but it’s still worth noting for robust deserialization.

[1]: https://github.com/go-restruct/restruct

[2]: https://github.com/jchv/pngextract

[3]: https://github.com/jchv/flsplit

DFDL ("Daffodil") https://en.wikipedia.org/wiki/Data_Format_Description_Langua... was made pretty much exactly for describing/parsing binary formats in a schema-like format. It is however based on XML and not JSON.

The author sort of answers the title question without appearing to realize it.

The answer is: "The things that exist are good enough for what people are usually trying to do"

It's unclear what the author's real use case is.

The reason proto/etc suffice for most people is that they do not care about the physical format, and control it. As a result, there is no need to define it explicitly or separately, and mostly people don't care to.

The folks that need to control the physical format are often those reading other people's formats. In a lot of cases, those producing the format don't care if you can read it or not :). They are often part of some ecosystem (say, photoshop, or whatever). As a result, they wouldn't use an "openapi for binary formats" even if it existed.

For lots of other kinds of standardized formats to read (video, image, etc), these days the processing is harder than the parsing. Not that it's worth nothing, mind you, but that i doubt you would find a lot of people care enough to think it's worth it.

You also start to get into fairly different kinds of formats quickly (IE network protocol formats vs file formats), and those differences often lend themselves to different abstractions (IE you often want to be able to stream decode network formats, but may care less about stream decoding file formats, etc).

The market that remains to be served well here seems pretty small. Certainly there are folks who want to control the physical format for some reason (whether they need to or not), etc.

But again, that feels like a small fraction of cases.

As a result, nobody felt enough of a need for OpenAPI for binary formats to push very hard on it. (and attempts at doing so in the past have failed for lack of value)

RecordFlux https://github.com/Componolit/RecordFlux

(author here, the blog is under an alias) oh, interesting! i hadn't seen this one yet. i think this one would fall under the category "transpile-first languages" (like wuffs or spicy); i.e. it is useful for portable parsers but faces obstacles for generating ancillary tooling.

Erlang has some special syntax for this and the people who use it seem to like it. Here’s a paper on adding binary comprehensions (like Python’s list comprehensions): http://user.it.uu.se/~pergu/papers/erlang05.pdf

Tangential to the rest of the conversation, but I often use/write binary templates [0] for 010 Editor [1]. The GUI-driven approach to navigating structures (such as PE headers) can be very useful.

I wish there was a similar tool that allowed me to simply highlight and bookmark blocks of binary data when I'm dealing with an unknown file format, such as a firmware dump for something exotic and/or proprietary.

[0] https://www.sweetscape.com/010editor/templates.html [1] https://www.sweetscape.com/010editor/

Swagger is, I think, named to reference SWIG, the tool everyone used to use to generate language bindings for C libraries, which is this.

SWIG is built on top of C .h files, which are the _real_ Swagger for binary formats.

Well, there's ASN.1: https://en.m.wikipedia.org/wiki/ASN.1

Welcome to hell, abandon all hope, etc.

(author here, the blog is under an alias) ASN.1 is in a similar category to protobuf: it doesn't allow definition of the physical representation of the data. (it does support multiple serialization profiles, but ultimately ASN.1 data can only be serialized in a few different ways.) you can't write an ASN.1 file to parse arbitrary binary formats found in the wild.

ASN.1's Encoding Control Notation[0] allows bit-level encoding control and can be used to write ASN.1 specifications for existing protocols.

[0] https://en.wikipedia.org/wiki/Encoding_Control_Notation

Can KaiTai parse a binary file/stream containing heterogeneous binary structures? Documentation seems not that clear regarding this. Also the structure defenition seems to needs compiling, what if there is change of structure at runtime? Can it handle it?

Is it possible to for an application to have editable structures and KaiTai to parse them without needing to recompile the whole application (I'm speaking here in a java context).

I've long looked for something like this for my day job, which involves implementing many different custom binary network protocols.

One new project in this space is poke:

http://www.jemarch.net/poke

It can describe arbitrary formats for both reading and writing, and it's usable programmatically, so I think it can solve the problem!

Maybe kaitai struct is what you are looking for? ;)

https://kaitai.io/

I did research into session types to try and solve this exact problem. However the theory still new and implementations are lacking. Again session types encode the logic, you still need some sort of structure/content language to describe the messages. Session types should be able to encode complex protocols and combined with kaitai, maybe something can emerge.

http://www.jemarch.net/poke

Another one from google: flatbuffers. We use this at work mostly to be able to be ISSU proof in out data models. It has some useful features for this like checking that the evolution of a data model is in line with ISSU compliancy guidelines, which we check in CI.

This interests me a lot. If blog author reads this, is there some way to reach you in pm?

that's me, please see my profile. :) (i've added an edit on the bottom of the article)

You got mail

you are looking for katai struct: https://kaitai.io/

katai allows you to declaratively define a binary format bit for bit - they even have an IDE

Can it parse a binary file/stream containing heterogeneous binary structures? Documentation seems not that clear regarding this. Also the structure definition seems to needs compiling, what if there is change of structure at runtime? Can it handle it?

Is it possible to for an application to have editable structures and KaiTai to parse them without needing to recompile the whole application (I'm speaking here in a java context).

Does COM count?

Hello DCE RPC, SOM, COM, CORBA,.....

Yeah... it's kind of amazing reading articles like this.

Protobuf, MessagePack?

(author here, the blog is under an alias) those are serialization formats, they don't allow for defining the physical representation of the data. for instance, you can't write a protobuf schema that de-serializes a .gif header.

What exactly do you mean by "defining the physical representation of the data"? Like, "here is a type called uint32, it is a 32 bit little endian unsigned number. Here is struct Foo, it has two uint32 fields". Like, that level of granularity?

Basically Katai struct but can also be used to serialize, not just deserialize?

It's SignalR or WCF with binary serialization!

gRPC

grpc is a layer on top of protobuf

Extending it to define transport concerns, akin to Swagger/OpenAPI.

In my view, gRPC is the closest "binary" analog to OpenAPI, which is a transport-layer concern first and foremost with discoverability features.

If your problem space is "a way to assign meaning to arbitrary bitfields", I don't feel that OpenAPI is a very good comparison. Its purview is "representing web APIs in terms of structured JSON/XML payloads". That is, OpenAPI also deals in logical and not physical representation. (unless it specifies UTF-8 or something - it might, I don't know!)

you know, i think you're right. on reflection, a comparison to json schema[1] (rather than openapi) would be more appropriate, as mentioned be defanor in another comment.

1: https://json-schema.org/

> In my view, gRPC is the closest "binary" analog to OpenAPI, which is a transport-layer concern first and foremost with discoverability features.

Well, it and every other binary RPC technology...

Shhhh....we are still in stealth mode.

Reply on news.ycombinator.com