Making our own executable packer, part 13, Thread-local storage | Hacker News Reader

Making our own executable packer, part 13, Thread-local storage

fasterthanli.me

58 pointslukastyrychtr23 comments

Great article. I particularly enjoyed the “beyond the basics” use of gdb.

I know the basics just like anyone else whose done some light debugging with gdb but it is hard to find information that goes beyond that. There’s a gazillion of half-baked gdb introductions but IMO not enough advanced tutorials.

Would someone explain what the primary intent and current best-case usages of thread-local storage are?

Is this more suited for certain languages than others?

https://en.wikipedia.org/wiki/Thread-local_storage mentions the two most common uses: errno (historically a global variable, but on modern Unixes thread-local) and having separate threads accumulate a value, to only merge them at the end, thus decreasing contention for the merged value.

> having separate threads accumulate a value, to only merge them at the end, thus decreasing contention for the merged value.

That sounds like just having scoped state in each thread that isn't global.

TLS lets you name that state with the same name across threads.

Yes, but sometimes, you can’t do that. For example, programming languages with a ‘parallel for’ may use a thread pool, reusing threads between iterations, but there won’t be scope to store the state in.

So, what you do in the loop:

  - grab thread-local variable containing local accumulator
  - if it doesn’t exist yet, create it and add it to the
    global list of all local accumulators (there’s a race
    here, but it will be rarely run)
  - do the work for the iteration
  - update accumulator

Then, after the loop, go through the global list of accumulators and merge them.

This explanation is a bit hand-wavy, but "correct enough" for our purposes.

Let us suppose that you wanted to store some per-thread data.

A naive way to do that could be to have a global mapping from thread ID to a pointer to your data. To look up your data, you would obtain the id of your current thread and look it up in the map. Of course, since that map is shared across threads, you'd also need to protect it with a mutex.

Operating systems themselves need to maintain per-thread data. When context-switching between threads, the OS will also switch out the current block of per-thread data. Applications can piggyback on this, so instead of needing to lock a shared map and then do a lookup to retrieve their data, it will already be available to them by virtue of the fact that it is essentially part of the current thread's context. No locking or global map lookups necessary!

It's super useful for many reasons:

  - to help make thread-unsafe APIs thread-safe

    E.g., errno, strtok(), and so on.

  - to store thread-private mutable globals

    (Thread-locals are more like thread-globals.)

  - to keep a reference/copy of global immutable
    data that might get replaced in the middle of
    a thread's handling of a request

  - to keep contextual data that you can't retrofit
    an API to pass explicitly (see also first item)

Pure Data made heavy use of thread local storage to turn the original application singleton (with lots of globals) into a threadsafe and reentrant library (libpd).

IIRC "jemalloc" achieves a speedup by using thread-local storage for (pointers to) bookkeeping and memory pools in preference to a heap shared between all threads - that makes it possible to do an allocation without using any synchronisation.

Why is he making a packer?

The better question is, why not?

> amos loves to tinker

Start at part 1.

No, he doesn't give you an answer, but you might just find it interesting enough to no longer have the qusstion.

I don’t understand what a ‘packer’ is - is it a linker?

From what I understand, it's something that 'packs' an application, either to preserve space or to obfuscate it (making it a little bit harder to reverse engineer it since you have to peek at the extracted contents in the memory instead of at the file contents).

So when you would run a packed application, there application would essentially start to unpack an encrypted/compressed blob of memory, and then jump to it once it's unpacked.

It compresses executables.

> I don’t understand what a ‘packer’ is

You've found the right series of articles then!

I read it - I did't find an explanation of what it was that he was building, just how he was building it.

Think of it of decompression software bundled along with a compressed version of the program you actually want to run. So execution starts in the uncompression part, it unzips all the code into memory, and then starts running the program you actually cared about.

See https://en.wikipedia.org/wiki/Executable_compression . They have been around since forever.

Packers were common in the 80's and 90's to save disk space. I mostly remember them from Amiga warez and demos.

The first few paragraphs on the first part explains it?

Executables have been fascinating to me ever since I discovered, as a kid, that they were just files. If you renamed a .exe to something else, you could open it in notepad! And if you renamed something else to a .exe, you'd get a neat error dialog.

Clearly, something was different about these files. Seen from notepad, they were mostly gibberish, but there had to be order in that chaos. 12-year-old me knew that, although he didn't quite know how or where to dig to make sense of it all.

So, this series is dedicated to my past self. In it we'll attempt to understand how Linux executables are organized, how they are executed, and how to make a program that takes an executable fresh off the linker and compresses it - just because we can.

Reply on news.ycombinator.com