Hacker News Reader

Comment by rui314 | Hacker News Reader

Author here. If you have any questions about mold/macOS, feel free to ask me as a reply.

Does mold support jobserver or some other mechanism to throttle threads?

We use lld currently but had to disable threading as sometimes in CI several of our test binaries would get linked at the same time. Whenever this happened the lld instances appear to have spawned enough threads to overload our Jenkins slave to the point that the master wasn't able to reach it and failed the build.

It's being discussed (https://github.com/rui314/mold/issues/117) but haven't reached any conclusion. The problem is that the jobserver protocol assumes that one process is one job, and its model doesn't fit very well to programs such as mold.

Thank you rui314. There is a lot of good information collected in that issue.

If you use Ninja, you can create a job pool for linking, separate from compiling, and restrict how many simultaneous linker jobs are run. You can even create and set job pools for Ninja through CMake, if you use that in your tooling.

Unfortunately GNU Make offers no such mechanism. And for Ninja build generation I don't think Meson does it either.

I was not aware of this feature and we do use CMake. Thank you for letting me know about it!

Can mold also serve as a replacement for dsymutil (i.e. linking DWARF info)?

If not, any plans to support this in the future? :)

I haven't thought about that that much, but it looks like dsymutil is a companion command of a linker, so the answer is probably yes, but I need to investigate it further to give you a concrete answer.

Someone was supposed to make llvm-dsymutil use the lld code, but that didn't happen.

Why is it so much faster?

The biggest reason is because it is multi-threaded. When building a program, the compilation step is parallelized (the build system invokes a compiler for each source file), but the final link step is not. So it is important to make the linker itself multi-threaded.

But even without multi-threading, mold is still faster than other linkers. I can think of various reasons why, but I don't know which attributes how much. I believe the biggest contributor is its efficient data structure -- it is hard to make program faster by writing fast code, but it can naturally be achieved by designing efficient data structures. That said, it is hard to compare two or more programs to find out why one program is faster than the others unless their designs are similar.

Note that recent lld is also multithreaded, and with tweaks, can be faster than mold at low core counts: https://bugzilla.mozilla.org/show_bug.cgi?id=1746462#c2

I wouldn't be surprised. In mold, if we have more than one choices to implement a feature, I always take the one that scales well for more cores even if it doesn't perform the best on low-core count machines.

My assumption is that future machines will have more cores than we have today on average, so I'm optimizing mold for such computers.

> My assumption is that future machines will have more cores than we have today on average, so I'm optimizing mold for such computers.

From the manpage of mold-1.3.0, I get the impression mold is designed to scale up to 32 cores, but not more.

  man mold | rg -C2 32
       --threads
       --no-threads
               Use multiple threads.  By default, mold uses as many threads as the number of cores or 32, whichever is the smallest.  The reason why it is capped to 32 is because mold doesn't scale well beyond that point.  To use only one thread, pass --no-threads or --thread-count=1.

Is this correct or does the manpage need to be updated?

Could you comment on which data structures are most critical to mold’s performance, and what makes them so fast?

Ah, I overlooked this explanation at first: https://github.com/rui314/mold/blob/main/docs/design.md

Thanks for the great write up as well as mold itself!

https://github.com/rui314/mold#why-is-mold-so-fast

Looking further into the future and the advent of io_uring in Linux, would you consider special-casing Linux IO ops to use io_uring or do you not expect any speedup there?

I'm not sure if io_uring can improve mold's performance, as it has to access random locations while copying file contents to apply relocations. Currently, we mmap all input files and an output file and use memcpy to copy file contents.

I see mold does not yet support LTO [1]. But if it were to, could we still expect performance gain over lld when doing LTO?

[1] https://github.com/rui314/mold/issues/181

mold does support LTO. It's not faster or slower than lld both in terms of link speed and the output binary's speed.

It might be asking for too much, but this project would be a nice addition to http://aosabook.org

Thanks for your good work! I still remember the O(n^2) complexity of ld.bfd when linking C++ code.

I want to write a book about linkers so that the knowledge I earned during the development of the lld and mold linkers wouldn't lost, but I don't have enough time to do that!

> I want to write a book about linkers so that the knowledge I earned during the development of the lld and mold linkers wouldn't lost, but I don't have enough time to do that!

I know this is a type, but it is interesting to see knowledge as a score or commodity you can "earn".

I'm just not a native English speaker.

> I know this is a type

( ͡° ͜ʖ ͡°)

I'm one data point but I'd buy it in an instant.

FYI: https://www.amazon.com/Linkers-Loaders-John-R-Levine/dp/1558...

yep yep I have it :)))

Your tweet shows quicker linking on a Mac yet you say down-thread that Mac executable linking is still not fully supported. So can we or can we not use it today on a Mac? (My main use-case is Rust.)

It can create Mac executables, but mold/macOS is still in pre-alpha and no one should expect it to work for their programs. Once it becomes out of beta, I'll release it as mold 2.0, so please wait for it.

Thanks. I will wait for an official announcement then. Just subscribed to release notifications on GitHub, too.

Do you have any plans for supporting other binary executable file formats?

We do support ELF (Unix) already, and we are working on Mach-O (macOS/iOS/watchOS/etc) now. Once Mach-O is finished, we'll be working on PE/COFF (Windows).

When are you going to start working on a 10x faster clang++? :-)

That's an even crazier goal which is probably 100x harder than writing a 10x faster linker. But I believe it's technically doable. At least, the world needs more crazy people who believe it is technically doable and take it as a challenge. If I get $$$ by selling the mold project to a big tech, I might be able to create a team with that money to tackle that crazy goal...

More power (and moneys) to you sir - I just used mold 1.3 to link qtwebkit right before this article appeared.

do you have a plan to supersede official linkers in gcc/ llvm?

I don't have a plan, and that's not what I can plan. They can plan in theory, but I believe that's very unlikely to happen.

if mold has multiple advantages over the official ones and no drawback, can't you send them an RFC once mold reaches 1.0?

None of gold, lld or mold can replace GNU ld entirely because they don't cover all features that GNU ld has.

Read the readme. Full compatibility with gold/lld is a non-goal and will significantly slow down the linker IIUC.

the whole embedded world uses linker scripts. for sure not mold goal. also, they don't have linker performance problems, but the big C++ apps have.

Reply on news.ycombinator.com