Would it be possible to create a binary library format that declares its own ABI? Has anyone done something like that in the recent past?
Not like GObject declaring the interface in a special comment (modulo typos). Something that's fully machine-parseable in any programming language and does not require access to the source code.
I mentioned C2FFI (https://github.com/rpav/c2ffi) in another post in this thread. That's an extra "spec.json" file that you need to distribute along with your library, but maybe that's a whole lot easier and simpler than inventing a new library format. And it's JSON, which is human-readable, so it's relatively easy to debug.