Mold: A Modern Linker
github.com
github.com
I assume part of the difference is that Zig has complete control over its environment, whereas Mold is trying to be a general-purpose linker, but still, I wonder if there's some insight to be gained from crosspollination there.
(like, maybe Mold could have an "--incremental" option with associate documentation explaining that this option is only beneficial if the input object files follow some specific guidelines)
On the other hand, by default, gcc or clang emits code that does not use GOT or PLT, which makes the situation much more complicated.
In addition to that, maybe you don't have to support all ELF fancy features if you know that you are linking Zig programs? I'm not familiar with Zig, but I can say that some kind of minor feature, such as weak symbol, can make incremental linking a lot harder.
When the Zig compiler is asked to produce an executable, using only .zig source code, it is in full control of the entire compilation process, frontend to backend, so it can make decisions that ease the requirements of its linker, in order to facilitate incremental compilation and linking. For example, when linking pure Zig projects, there are no relocations; the code is generated directly in place in the final executable, almost as if there is no linker step. However, when asked to link against third party objects or static libraries, Zig must compromise some of these advantages. Currently, in this situation, Zig will fall back to doing the incremental compilation of .zig code into an object file, and then invoke LLD (via the zig executable invoking itself as a child process) to link the objects together. As the Zig self-hosted linker gains functionality, this fallback will happen less often; instead the compromise will be in the code paths taken in the linker, based on what assumptions it can make about the linking requirements that are required for a given job. The long term plan is to eliminate the dependency on LLD altogether.
Side note - mold is a brilliant project! Thank you for making it and pushing the state of the art forward! Also I love the logo.
[0] https://github.com/ziglang/zig/issues/1535#issuecomment-6784...
It’s probably fine to not avoid most Unix APIs but fork() is truly an exception here. Fork() is not friendly to any third party library you may use because their state may become invalidated after a fork but the library has no way to know if the process has been forked. This is especially bad if the library uses multi threading. The best way to avoid a random unintended bug later on is to only use fork() when you plan to execve() right after.
I wanted to keep a linker process running as a daemon so that it doesn't read the same files over and over again. After loading input files, the linker becomes a daemon and calls fork() to create a worker process. Then the worker process does the rest of linking. In other word, a daemon is a "clean" copy of a linker process image, and each child is specialized for each actual linker invocation.
It turned out that the linker runs much slower with fork() because of the overhead of copy-on-write. You cannot keep a fresh copy of a process just by calling fork() for free. There's a tax associated with it. I tried to workaround, but in the end I had to give up with the fork()-based worker process design.
Honestly that's a good thing, or your program just fundamentally couldn't possibly ever work on native Windows!
Modern mainframe development environments are browser based.
Either way, threads should be controlled by the application and not libraries. Well-written libraries like sqlite and Lua are parameterized by I/O and concurrency. They don't read files and start threads behind your back.
That kind of thinking comes from node.js and Rust as far as I can tell.
There is never ever a situation where a program like GCC or Clang will acquire a third party dependency without a commit that explicitly changes the build system.
The Python example isn't really relevant because Python modules are shared libraries, but a linker doesn't have or need shared library plugins (i.e. dynamically linking against arbitrary machine code at runtime).
First, pkg-config exists and most projects on Unixy OSes use it these days. It doesn't really make much sense to argue "it all has to be specified on the command-line" when the command-line the developer cares about looks like this:
gcc `pkg-config --cflags gtk+-3.0` -o test test.c `pkg-config --libs gtk+-3.0`
(That adds things like `-lcairo` which aren't at all visible at first glance)
Second, why do you think things like libwayland-client, libfreetype, and libexpat show up in the ldd output? They're certainly not in the pkg-config output.
Likewise, nothing says a library has to bump its major version number when it adds a new backend dependency, which means that a distro may add a transitive dependency through a shared library update that doesn't correspond to you bumping your version.
(You can gain new backend dependencies without even recompiling anything you're responsible for.)
Heck, this post explicitly touches on that disconnect between what you add to your build plan and what ldd lists for C and C++ codebases.
(if it's unclear, this is strong agreement with chubot. If you make the problem harder than it has to be, you have only yourself to blame...)
I honestly thought that was the meaning the author was trying to evoke before I saw the picture on the github page.
I'm not sure if that image is the best way to communicate that status, given that the sudo sandwich logo exists (which coincidentally bears some resemblance to your moldy bread). A big bold "not ready for production" at the top of the README is probably a better way to achieve that.
Which I think might just be the greatest name for a new linker.
EDIT: You may just have to settle for speed/parallelism being conveyed by 2..3 spigots pouring in. :-) It's perfect - you can stay with moldy bread while it is a major work in progress and evolve to the more finished logo when your own work is "hardened" -- all without changing the name. ;-)
[1] https://www.foodandwine.com/cooking-techniques/baking/best-b...
M comes after G so the name tracks perfectly while also being weird and distinctive.
Looked it up and realised that the US spelling is actually also 'mold' https://www.oxfordlearnersdictionaries.com/definition/englis...
I've just been interrupred while writing this to be told that America and Britain number calendar weeks differently (Britain follows ISO[1]) and that Apple's calendar is fixed to the US version. It never ends…!
Huh, apparently so! So Rust-the-language isn't necessarily named after oxidized metal, but it _is_ named after something that's named after oxidized metal.
> <graydon> fungi are amazingly robust
> <graydon> to start, they are distributed organisms. not single cellular, but also no single point of failure.
However, I will note that graydon uses words like "I think I" and "I remember kinda" and everyone says "yes this means this is 100% the source of the name" whereas I take it to be like, "this is one of many reasons that Rust is named Rust."
You also have to remember that Rust was a very different language in many senses back when Graydon was choosing the name, so allusions may not make sense now but may have then. Early Rust was much more erlang-like, which may make the above feel more relevant.
Edit: Had fun thinking up a couple others:
"whimsically eclectic logical design"
"willfully egregious lackadaisical decision"
"wise element location director"
"world exploring layout detector"
"wrong-headed exasperating liability defender"
"workaday execution layer developer"
"wonderfully elegant logistical delegator"
Those performance improvements are crazy as well: here's hoping this becomes the standard.
Is this safe? Are you sure that the _exit() delays are not part of the kernel committing all the pending mmap I/O to the buffer caches? If a build script links a binary using this, and then immediately executes it, the second (background process) might still not be finished. Is it guaranteed that all of the mmap I/O will be visible - or will the binary appear incomplete?
I don't know the answer to this myself - it would be clear cut if the linker was using write() calls, because UNIX guarantees that future read() calls will see the results, but the ordering guarantees with mmap I/O are far more loose, I believe.
That being said, I could easily believe that largeish pwrite(2) calls would be comparably fast compared to mmap, since mmap needs to play with page tables, and page faults on x86 are expensive. MAP_POPULATE would also be worth trying if you’re not already using it.
I assume that copy_file_range(2) is out of the question due to relocations.
It is usually much better to have multiple regular processes, instead of threads, that only share chosen mappings, if you want to use munmap. Or, you can terminate and join all your threads before you start munmapping.
[target.x86_64-unknown-linux]
linker = "ld.lld"
Why do we need to copy data at all ?
Can't we just have an object file with pointers to other files ?
Like sure, if I ever want to ship my binary somewhere else, I'd like to do this. But for local interactive development, all the files are already in my local machine, so I don't know why we would need to create a second copy of them within some other file.
What is more interesting is the real bottleneck was not about mmap vs write(v) but in the filesystem. If you create a fresh file and write 2 GiB of data to that file, it takes like 700 milliseconds on my machine (ext4 fs), but if you write the same amount of data to an existing 2 GiB file, the IO speed doubles. So, the filesystem's performance to allocate new disk blocks seems to limit the performance of my linker. That reminded me of the axiom: don't guess about performance but measure.
If you know the total size ahead of time, you might try using fallocate(2).
You can give each thread its own malloc that just mmaps what it needs, if you are leaking everything anyway.
This is a case where using a raw new() and raw pointers is much better than using a smart pointer, because touching things to run destructors unnecessarily is itself expensive.
My experience with every linker so far is that to have XO I will need to specify the memory layout in a linker script manually. It's not as nice as simple linker argument, if that is possible.
If so, that's a relatively new CPU security feature. I think some ARM processors support it, but AFAIK x86 doesn't support it at the moment. On x86, if you make a page executable, it automatically makes the page readable. R and X bits are not separated in the page table. I bet Intel and AMD will ad NR bit (no read bit - analogous to NX bit) soon, though.
If they really support it, I didn't pay too close attention, but at least it's on their mind.
Can this one statically link shared libraries? That would be a great feature.
EDIT: I realize that such a feature would necessarily involve a fair amount of black magic. But it does not seem an impossible endeavor.
Well, an .a file is just an archive of object files. Turning several object files into a single binary is pretty much what a linker does. However, you can't really automate this because object files might have external dependencies and the linker needs to know what these dependencies are.
If it can be done at runtime of the program, I guess it can be done at runtime of the linker, doesn't it?
EDIT: also, building with unresolved symbols requires the host application to know about and link all the required external libraries, which is usually not what you want...
static Symbol *intern(std::string_view name) {
static ConcurrentMap<Symbol> map;
return map.insert(name, {name});
}
static ConcurrentMap<Symbol> map;
static Symbol *intern(std::string_view name) {
return map.insert(name, {name});
}
// __start_ and __stop_ symbols
for (OutputChunk *chunk : chunks) {
if (is_c_identifier(chunk->name)) {
start(Symbol::intern("__start_" + std::string(chunk->name)), chunk);
stop(Symbol::intern("__stop_" + std::string(chunk->name)), chunk);
}
}for (std::string_view arg : config.version_script) parse_version_script(std::string(arg));
Isn’t ‘cat’ io-bound?
> It is because most objects that are allocated during an execution of mold are needed until the very end of the program.
Using free would not affect that memory consumption much because most consumption would be freed at the end of the program.
> Since the missile will explode when it hits its target or at the end of its flight, the ultimate in garbage collection is performed without programmer intervention.
[1]: https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...
Now the author seems a smart fellow, so maybe he did just that already, I didn't check the source.
The fact that someone does it wrong doesn't mean that we should do it wrong as well ;(
I don't think I like this approach. It may work now, but will probably seriously limit the possibilities in the future.
The fact that this program can successfully link Chrome means that we have fairly solid baseline performance metrics we can use for "big" programs. Chrome is just about the largest program you might ever need to link.
Because later if you want to reuse parts of the code in a continuous environment (e.g. a daemon), then you will be surprised that you have memory leaks all over the place (or worse, someone else will discover it by accident).
I don't have a problem with the end-of-process-releases-all-memory optimization. But I had the impression that the author uses let's-worry-about-leaks-later-because-OS-takes-care-of-it-for-free-(in-my-use-case).
Best approach to take would be to create a memory pool with fast allocation (e.g. TLAB allocation in Java, or how computer games do it), in order to have control over how the memory is freed or when.