How is a binary executable organized? Let's explore it (2014)

I assume there's a better modern source (assuming for some reason you don't want to reference libbfd &c, but really, you at least want to cross-check with it) but BITD there was an AT&T System V book - I think it was https://books.google.com/books?id=mrImAAAAMAAJ but only at about 70% confidence, it's part of a series and might have been one of the others - that was "arcane but true" for ELF on existing platforms (at the time, which was the mid 1990s, which is why I hope there's a better starting point now...)

I agree it’s very nice. I’d also like to know how it was done.

Also if you click two bytes that are in the same caption group one after another then it bugs out.

I wrote it all by hand :)

Lately, I've been using Svelte for interactive visualizations (see my post on using a tool called Astro with Svelte: https://avikdas.com/2023/12/30/interactive-demos-using-astro... ). But this one is all hand-written JS!

Thank you! My passion is teaching, and I created interactive visualizations for a CS theory class I taught: https://cstheory.avikdas.com/

If you're interested in collaborating for your classes, reach out to avik at avikdas dot com

> reserve some global array in the executable

This is neat but has a limitation in that it cannot be expanded after the program is compiled and linked. Resizing the array would invalidate all pointers that follow as well as render incorrect any code that takes its size.

This can be solved with a layer of indirection: just append the data to the executable and write its size and file offset in the array. That way the data block can be freely resized. That's the solution people told me to use and indeed the one that I usually see in existing repositories. The problem is you run into some additional complexity later which results in the loss of portability and thus the main reason to choose this method.

> it can open itself

That's the crux of the issue. How does an executable open itself? That's where portability goes out the window. I've seen source code that opens argv[0] which is under the control of the parent program and therefore unreliable. I've seen code which opens /proc/self/exe which is Linux specific. I've seen code that calls Win32 API functions to get the path to the executable. All this just so it can open and load into memory a file which the kernel has already loaded, just so it can read some additional data off of it.

My solution sidesteps that question entirely. It just adds a LOAD segment for the embedded data which instructs the kernel to map it in automatically before the program even runs. There's no need to open, seek or read anything, it will already be there by the time execution begins.

The auxiliary vector contains a pointer to the program's segments table so it can reach the data from there. Then it's just a matter of walking this table looking for the custom descriptor segment. It's all done in a structured way, using the standard magic number locations and ranges. There's no chance of a magic number being recognized by mistake.

The only possible portability issue is the availability of the AT_PHDR, AT_PHENT and AT_PHNUM entries in the auxiliary vector. I'm not sure if they're standard. I know Linux has them and it's all I personally care about but if these entries do turn out to be standard then I can confidently say that my method is portable to any ELF-based operating system.

Put the offset at the end! Famously, the table of contents of a .zip file are the end rather than the beginning, which has many useful properties (such as being able to patch the contents by only appending to an existing file). And you can concatenate an executable and a .zip and get a file which is both.

The entry point can be anywhere in the .text section, and often won't be at the beginning of the section.

> Furthermore, such initialization is thread safe so that one initialization happens despite multiple concurrent calls of the function.

IIRC, there are some popular compilers in which initialization of static variables inside functions is not thread safe (even though AFAIK the C++ standard said they should be).

The issue with Meyers Singletons is that every time they're accessed a flag must be checked first. This is bad in hop loops.

Wow, great read. I worked on the Windows DLL loader and we had to implement similar mechanics for similar reasons. The PE image format makes some part of this a little easier, but the complexity is essentially the same.

> Does that mean this doesn't work in a freestanding environment? Yet another reason to assiduously avoid global variables. I suppose that's why I never ran into this issue.

Why do you say that? A freestanding environment will still enter through _start and execute the compiler generated code to initialize globals before main. What I can’t recall is how compile time non-0 values are initialized - I think it could be part of the bss and initialized by the loader instead (but freestanding environments would implement that too as part of being a target for a language) but both them and runtime initialized globals initialized between _start and main would work.

Basically freestanding targets might not give you access to runtime APIs (eg POSIX) but the language is still the language and all features defined as language features should work and it’s the responsibility of the compiler and target environment to provide that guarantee.

You probably know this, but in C, if you wrote the comparison like this

  if (password[0] == 'Q'
   && password[1] == 'w'
   /* the rest of the letters... */
   && password[8] == '3'
   // C strings are 0 terminated
   && password[9] == '\0')
  {
      ...
  }

It will probably compile to something like this on x86-64, assuming a "password" is a pointer held in RDI (although I didn't look at a compiler's output):

  cmp byte [rdi], 0x51 ; 'Q'
  jne false
  ;; the rest of the letters...
  cmp byte [rdi + 9], 0x0 ; '\0'
  jne false

The benefit is that the strings utility wouldn't see the string "Qwerty123" anywhere in the binary.

Unicode and bounds checking code probably complicate this generated assembly in other languages.

It would be nice if there were some way to write a macro or an inline function so that you could write a comparison like

  if inlineStringEqualP(mySecretPassword, "Qwerty123") {
    ...

And then it would expand into something like the assembly I wrote above, so that way, the string literal "Qwerty123" isn't embedded in the final binary executable anywhere. I bet it's possible with C++ templates somehow, but it would be messy.

It's even quite interesting to run strings on programs I wrote and I do it regularly. For example it can be faster to get the version string from an .exe using strings if you know what to grep for than to run "wine program.exe --version".

Not in general.

They don't. APE loads a different syscall table depending on the platform. Or something like that, I don't claim to fully understand the internals

Don't tell jart this or she'll probably add it to cosmopolitan