Evolution of the ELF object file format

(Personally, that makes me a bit uneasy: for instance, if the copyrighted spec lists the file sections in a certain order, and your implementation happens to output them in the same order even if it doesn't have to, then have you infringed on the owner's copyright of that particular arrangement?)

Meanwhile, they could have gotten a patent on (some parts of) the format, but that doesn't seem to be the case here.

Has anyone looked at creating a new object format for Linux? A non-open spec seems a minor issue, really, in an era when we put binary blobs in the kernel (hi Nvidia.) But the more decades I work in closed source, the more I value open source, and believe keeping _everything_ open.

https://wiki.osdev.org/ELF#External_Links

* ISA manual * ELF (generic ABI, psABI (e.g. x86-64-psABI), OSABI) * DWARF * Floating-point * Language standards * Itanium C++ ABI

and probably a few other stuff.

> it seems like my failure to find the most up-to-date specification is simply because one doesn't exist.

While the up-to-date specification is unavailable, hopefully it is not too bad because all essential features have been completed as of 2001:)

RELR is a linker and loader feature, not on the compiler side. Compressed debug sections are a pretty natural extension.

> Second draft published May 3, 1999. > ... > New dynamic section tags DT_RUNPATH and DT_FLAGS added. Dynamic section tag DT_RPATH moved to level 2.

In glibc, DT_RPATH and DT_RUNPATH have different semantics regarding precedence related to LD_LIBRARY_PATH. In FreeBSD rtld and musl, DT_RPATH and DT_RUNPATH are equivallent.

I considered using them. Ended up not using them but they did inspire the actual solution.

At the time I wasn't able to find much information about those things and how they're implemented and used in practice. Couldn't be sure if they were a good fit for what I wanted to do.

The best functional description of these notes I've ever found actually came from some old Linux standard base document. Apparently people would use their company's name as the note's name and enumerate a bunch of metadata types that are specific to their company. To me that sounded a lot like the "email the guys in charge to ask them for their blessing" situation described in TFA. I'm just a random guy and I just wasn't gonna show up out of nowhere in somebody's mailbox asking for some official designation for my hobby language.

There was no practical reason to use these things either. Turns out the kernel doesn't care about PT_NOTE segments at all. They don't get loaded into memory automatically. I'd still need to add a PT_LOAD segment for the data even if I had used this stuff.

The PT_LOAD segments are the only ones that really matter. Those are the segments the kernel cares about. Ideally I would have been able to simply use a PT_LOAD segment for the data and nothing else. When I tried it though, I discovered they had two limitations: alignment requirements and lack of type information. They have to be aligned to some page boundaries which obfuscates the true size of the data they contain. I also can't easily tell them apart from each other, I'd need to put some header in the data itself in order to find the correct segment which risks false positives.

Those two problems are solved by the PT_LONE segment. It describes the true size of the data block inside the PT_LOAD. It provides a unique magic number for finding it while iterating over the program header table. No possibility for mistakes. When it's found the program gets a pointer and a size and the data is already in memory because it's contained inside a properly aligned PT_LOAD segment. This makes the kernel do a ton of heavy lifting before the program has even begun execution.

This is what the PT_NOTE segment boils down to anyway. It's just a memory area. The standard just happens to mandate the presence of a header structure on those segments. The PT_LONE segment doesn't have that requirement so I just replaced that opaque binary header with S-expressions that my interpreter simply parses instead. Much nicer in my opinion.

> you can always leave the section table in as well

That was the first thing I tried. The objcopy tool even has options that let you embed arbitrary files inside the binary by putting them into a section. Seemed incredibly easy at first.

Turns out that ELF sections are irrelevant to the kernel: it only maps in the PT_LOAD segments. Had to ask lots of questions on stackoverflow and read the Linux ELF loader source code to figure this stuff out. Then there's the fact that linkers don't generate PT_LOAD segments covering the program's sections. Apparently only the dynamic linker uses that stuff so there was never any reason to make the kernel load it. It's just how things are done and it doesn't look like things are about to change to accomodate other use cases.

Since this is not done, anyone who wants to embed data into the ELF has to somehow find and open the running program's own executable, read the thing into memory and parse its contents a second time. This is just bad on so many levels. It's fragile and duplicates work the kernel has already done.

The proper solution is to make the kernel load the data as part of the program's memory image. That requires editing the segments, not the sections. Tools for editing sections are quite featureful. Tools for editing the segments didn't actually exist.

So I made a little tool to patch this data into an ELF file. This turned out to be very hard and annoying. There were two possibilities: either move the table to the end of the file, or reinvent the linker. The first option leaves a huge hole in the file where the original used to be but it does allow adding entries without relinking everything. The second option doesn't waste space but is orders of magnitude more complexity than I had signed up for.

Took a third option by asking for linker support instead. The mold maintainer added a couple lines of code and suddenly my ELF files had PT_NULL entries tailor made for patching which is just awesome and the best possible solution.

I asked GNU ld maintainers to add this feature as well. They suggested I try using linker scripts instead. I had actually tried that before emailing them and it's actually what motivated me to just ask for this. I seriously hope I never have to write linker scripts ever again.

I opened an issue on the LLVM issue tracker but never heard back. I also tried to implement it in the patchelf utility but couldn't figure out how it works.