Executable Is a SQLite Database
fzakaria.com
fzakaria.com
https://www.sqlite.org/vtablist.html
You can "mount" your filesystem (or anything else) as a SQL database, wtf. That's amazing.
This sounds like it could be extremely useful.
The key thing here is to be able to discern between real and manipulated input information. For this, gut feeling usually works best: it spots that whiny, attention-insisting, importance-seeking, fear-inducing tone of a manipulator, but oftentimes the higher-level nervous system of the brain suppresses that signal (e.g. "How can a well-educated and charismatic person feel a bit off and responsibility-lacking sometimes? It cannot be right, so it must be something with my interpretation of reality. I bet they have the best intentions.")
An even better approach may be not to search for feedback at all, unless it's shared naturally without any strings attached.
If I put on my academic-paper-reviewer hat, I’d probably ask: what does that insight get us? What knowledge can we pull from databases into compilers? If we convert compiler/linker/loader operations into database lingo, then do we see patterns or more insights that let us resolve existing controversies/problems?
I haven’t read the author’s academic paper, so I don’t know what he considered the contribution to be. The contribution is “what knowledge was learned that improved the world”. The insight is enough for a minor paper, in my opinion. But if you can show it is the key to tackling an existing problem, that would be a great paper.
Academic reviews are rough. Hasty, inconsistent, often poorly written or incomplete. Responses can be random with good papers getting harsh reviews. Don’t take it too personally. Sometimes the information in a communication isn’t in the content - it is what it says about the communicator. Keep working on your writing and thinking clearly and your odds of better reviews improve … but will never be 100%!
https://sqlite.org/draft/appfileformat.html
There is little reason that ELF needs to be so complex. My motivation was to bundle multiple wasm modules together along with data.
It works great as storage for geometries and attributes and at the same time you have the full power of sqlite for all kinds of analysis.
I enjoyed all the papers I read on my graduate classes that bordered on "art" whereas now they seem to be obsessed with performance only.
This has been on the back of my mind for a while. And, as other commenters have noted, it would be great for the file to contain the (self-modifiable) Lisp image, a builtin virtual file system, and whatever the application want to use as (runtime modifiable) extra tables.
I find SQLite dynamic linking being basically compatible with ELF dynamic linking to be very impressive, I can imagine that if well done, it cloud replace most uses of AppImages with a much more efficient format, like the author suggests.
How about an option for compressing section contents within the SQLite blobs, since the author mentioned you can't mmap directly the text pages and have to copy anyway ?
There are two extensions that I was thinking would make a SQLite executable truly unique.
First, a linking extension that would allow patching in functions and hooks more directly to allow for a very powerful plug-in system. Imagine the plug-in SQLite defining a BEFORE/AFTER/REPLACE hook for some symbol the host SQLite defines as extensible.
Second, re-linking at runtime. This would require application author cooperation because you won't be able to do that from anywhere, but imagine changing a dependency or loading a plugin at runtime through editing the db, and the interpreter just maps that on demand/automatically in the background, now next time your web server accept(), it calls the new version of the handling function.
I got another one: bundling multiple (completely separate) executables with shared dependencies in the same SQLite, selected by argv[0] when called, like what busybox does
Which makes inability to punch wholes in the middle of the files without having to read/write huge chunks of data quite annoying, especially when you write a linker or an archiver. Reserving enough space for the header is quite non-trivial.
Sounds like a great use case for:
1. ELF file to SELF file
2. modify SELF file
3. SELF file to ELF fileLike, it's an ugly spec, but it's clear enough on ordering/layout that I think checksumming is probably tractable to get parity for.
If the author wants to make a case for including schema metadata in an object file, again why SQLite? This strikes me a lot as someone who is trying to find uses for their favorite hammer (a very useful hammer I might say) rather than a serious exploration of what a new, improved object file format would look like.
(And that’s totally ok)
sqlite would have made this pretty trivial. Replace the .text fields of a few rows. Insert a few rows for symbols, update a few from the old object.
The best part - there’s tons of library support for sqlite. If it was a new object format, there would be no support and I’d just have to write different parsers and generators.
The schema might the the biggest issue for efficiency. As others have noted, ELF->SELF->ELF might be the best use case for compatibility. That said, a big part of [0] was performance, and I don’t know how sqlite would have done compared to my naive elf parsing and manipulation.
Would this normally be something the compiler would handle, or would it be the linker? I guess I’m curious how this would impact any optimizations the compiler implements.
Also I’m guessing for your use case (elf edits) you would be looking at both SELECT and INSERT type operations?
This would most like just be the concern of the compiler and assembler. In GCCs case, my understanding is it just inlines the assembly into the output assembly stream and the assembler is none the wiser. The issue is that MetroWerks doesn’t support inline assembly at the top level (for defining new function symbols), but only inline within the context of a function. That means some additional setup and tear down that shouldn’t be present may be injected instead of the raw assembly.
There’s possibly some upseet potential and the idea that it could be done in place could avoid loading the entire thing k to memory (though the files are small for modern systems).
For my own edification I often let performance & size slip to see if I can re-imagine something simpler from the start from a complete new direction;
This is often the greenfield hubris because it doesn't have all the same battle-scars of the full implementation so I try to make sure I take a very different approach to make it worthwhile :)
Then I just have faith that performance and size can always be improved over time.
All of the benefits of SQLite disappear once one diverges from the format in any way. At which point it would be better to ask: "what is the best first-principles implementation of this idea?", instead of "what is the minimal change to SQLite to achieve this specific narrow goal?"
> Surprisingly a lot more falls out as well: .dynstr is gone, because name is TEXT and SQLite already interns strings
What's with this claim? SQLite does not intern strings, as far as I can find anywhere, and a quick test shows that duplicate strings are actually duplicated in a database file. You can intern strings in SQLite manually with an intern string table, but it doesn't happen automatically.
Let's go deeper. it's a webserver app + the server code + application code + db, so pocketbase++ where it's also the deployment target.
Then combine it with APE liek system, and the same file loads and stores things on every platform. evil laugh
Very cool hacking! My hats off to the author.
A version based on this which carries around an overlay filesystem would be comparatively straightforward, the hard work is already done.
BIOS -> Database that has everything including OS.
This way we get WORM and AI integration.
Isn't this the whole point of executable file formats? To avoid loading everything all at once, share immutable segments with other processes, be extremely fast when loading, and integrate with OS page fault handlers to load data on demand?
I don't understand this project at all.
Even if it's possible to query the executable using SQL, we can always just use the SQL frontend to interface with executable data. I mean, ELF as a backend format for the "executable database". Like osquery interfaces with the system using SQL language. Osquery doesn't convert all OS data structures to sqlite :)
Btw, I think SELF is already "coined" by Signed ELF executables - SELF files, existing for example on Playstation 3.
OK, in '70s, '80s or '90s when compute and storage resources were limited and every bit counted, specialized formats did make sense.
But nowadays we'd save enormous efforts by just packaging stuff in SQLite databases. Microsoft's proprietary file formats (Office, Power BI etc.), OpenOffice/LibreOffice OpenDocument format, or almost everything else would fit perfectly.
Documents, files, are all data (called "data files", aren't they?), including executables, which as can be seen in this article are also databases.
(Which is to say a lot of the specialized formats didn't exist just because of compute and storage resource limitations, but also because training/research/standards work came later. Plus complications like software patents. ZIP files have been around since the 1990s but ZIP only became a common general use container format for "everything" almost exactly 14 years later when patents finally expired.)
OR you restructure the code and data to work around the page headers.
The author instructs the OS to execute it natively my adding it as a binary format.
I think this is awesome though, feels like a lot of things on OS's could just be represented by databases - where does it end?
My mainframe experience was on an IBM 370/158 using VM/CMS, and I can't see any real connection between the OP and its executables.
I suppose there's a very vague connection to how the LOADER could load TEXT files (similar to a .o file) into virtual storage, without having linked an executable (a MODULE). But that's about runtime usage, not file format, and there's certainly no SQL functionality that applies to executables on mainframes.
- osquery: virtual table abstraction over proc fs. https://www.osquery.io/
- nushell: shell with structured I/O. https://www.nushell.sh/
A flat namespace might actually work with an app-based OS model (like on android) where everything that happens is associated with an app ID and apps mostly can't talk to each other. But there's a reason nerds don't do nerd stuff on smartphones.
Then there was Microsoft WinFS, which promised much but got killed in ~2003. A shame.
One nit-pick;
> The preload table is a list of objects to map last, so their exports win.
made it sound like the 'map last' is a consequence of this new perspective, but it’s really still a loader convention: by definition, the symbol-resolution query needs to give entries in the preload table precedence over ordinary dependencies.
I’m wondering if the anti-clickbait filter could benefit from some lightweight LLM integration. E.g. when the submission hits the front page, if the title was “de-clickbaited” and wasn’t changed manually yet, run it through a cheap model and see if that was the right call.
That would preserve the non-clickbait meaning of all such titles I can spontaneously think of.
SELECT * FROM files p LEFT JOIN files c ON p.id = c.parent_id LEFT JOIN files gc ON c.id = gc.parent_id LEFT JOIN files ggc ON gc.id = ggc.parent_id .... where p.id = 123
Not the XML protocol, that's annoying (although usable) but the structured data model.
The “Inception” graphic was hilarious and made me wonder if perhaps the site itself is served by some SELF binary (is it?? that would be insane)
(Even with ELF, a kernel can have a relatively simple parser and loader and leave more challenging work to user space... like relocations, dynamic libraries, etc.)
Whats worked for others?
ha ha ha
yes
Even more broadly applied: Every base of data is already a database (that's what the compound word means). Programs like sqlite and postgres used to be called by the more precise term "Relational Data Base Management Software" or "RDBMS" until their use became so widespread that they were colloquially called databases instead.
The very first data base structures were more like, each geometric sector of a hard drive is a record, and each head is a column. This was a straightforward translation of punch-card workflow onto a magnetic disk.