Mergeable libraries [video] | Hacker News Reader

Mergeable libraries [video]

developer.apple.com

165 pointscovector87 comments

It's really weird to me that they're talking about balancing static vs. dynamic in terms of the dev-end build time speed vs running the software load times.

To me the dynamic vs static balancing act is about compatibility vs security where dynamic linked applications easily get lib security updates but the binary may or may not work with your system libs and statically compiling is more compatible but doesn't automatically get system lib security updates.

They are talking about app-bundled dylibs, which cannot be updated separately due to the whole-bundle signing.

In other words it's a fix for a somewhat Apple-specific (and in this case Apple-created) problem?

iOS definitely isn't the only platform where apps ship their own versions of DLLs.

No, not the only one, but the converse isn't rare either. Whether it's "bundles" or containers, a lot of people act as though bloat and slow build times because of static linking are inevitable when in fact the pain is self inflicted.

The problem is not Apple-specific and their solution could be useful elsewhere.

The specific optimization this achieves is during build time only: these new files are static libraries that are quicker to link. It is a small shift of some of the linking pipeline into the (parallel) builds of dependent libraries, rather than heaping it all onto the linker at the end of the build. The linker has to essentially re-link from scratch for every small change.

Parallelization has long been known as the best way to speed up linking. This Apple change comes in addition to rewriting their linker to be massively parallel. Mold did it first, though: https://github.com/rui314/mold

edit: lld did it first, mold improved upon it

A faster development cycle is one of the coolest improvements that Zig brings – lightning fast recompiles, bringing the native development cycle closer to web speed. More languages should get this capability, and Apple needs it for Swift to get instantly updating SwiftUI previews.

Static linking is required to get the best performance: no linker runtime cost, cross-language LTO and dead code elimination

If this optimization is generally applicable and developers find it worthwhile, I could imagine this making its way to GCC-land. At the very least, gold needs replacing/rewriting to be competitive now.

edited: for clarity

> Mold did it first, though: https://github.com/rui314/mold

Before LLD?

I had forgotten about LLD, you are right, but also, they’re from the same author! :P

lld has parallelism here and there, as well as ld64. I do believe ld64 is technically the first linker that started parsing object files in parallel.

Parallel linking was conjectured impossible for a while without breaking static archive semantics (AB-BA problem). Michael has a good explanation in https://www.youtube.com/watch?v=ONGVraXbJOo (minute 3 and beyond).

> Parallel linking was conjectured impossible for a while without breaking static archive semantics (AB-BA problem)

Didn't Tera/Cray have a parallel linker back in the 90s?

Whole-bundle signing wouldn’t need to prevent security updates of individual libraries to be loaded when provided on the side, similar to cryptexes.

I think the whole security thing of relying on dynamic linking is a red herring.

It is neither necessary nor sufficient. It will miss private dynamic libraries, Docker containers, potentially virtual envs.

And it has a huge cost in terms of maintenance and backwards compatibility.

In a world where we have regular builds from source, this model really doesn't have a place.

> It will miss private dynamic libraries, Docker containers, potentially virtual envs.

Yes, it will. Does that make dynamic libraries a red herring, or does it mean these approaches all make the same mistake? Please think before you answer. Just because something's common doesn't mean it's wise or necessary.

> it has a huge cost in terms of maintenance and backwards compatibility

How much are those "huge costs" unique to that approach? Maintenance and backwards compatibility are still problems even with static linking or embedding into containers or any other alternative. It's disingenuous to count common problems against only one approach. The real tradeoff is between problems that are unique to each, and it's possible to disagree about which choice is best without stacking the deck for one side.

Ultimately if the app is unmaintained, then it's likely to have security issues even if it's using dynamic libraries. If it is maintained then it shouldn't be a big deal to update it.

Security is not black and white. Better to be able to fix some security issues of an unmaintained app than none.

This is a bit of a tangent, but I don’t think it is sustainable in the long run to require the ever-growing volume of software in use to all be actively maintained. We should find a model of software development where we can achieve sufficient security without having to maintain and rebuild the world all the time.

Long term I think we'll get most of the way there by eliminating C and C++ from our software stacks. Most security vulnerabilities (and even more of the most serious ones) are memory safety or UB related. And systematically eliminating those will give us more time to audit our code for other issues. Eliminating C and C++ also likely leads us to standardised build systems which makes rebuilding things much much easier.

I disagree, especially for the desktop. I think there's great value in being able to update a shared library and get a fix in place for all applications, versus waiting for each application to update.

Apps ship with vendored or statically linked dependencies everywhere possible these days because system wide updates of shared libraries have caused too many breakages. It's just not a viable technology for todays' ecosystems.

> It's just not a viable technology for todays' ecosystems.

I disagree, obviously. The problem you cite is very real, of course, but there are numerous ways to address it.

I think that applications that avoid shared libraries do so because that's the path of least resistance.

There really aren't numerous ways. You statically link, vendor your dependencies and bundle your app, specify version numbers for your .so dependencies, or use symbol versions.

The latter two are the only options that involve any sharing of libraries. And avoiding it isn't just a path of least resistance but one of reliability and reproducibility. If I cannot pin the dependency down to the exact bytes I tested it with, I cannot tell you if you should trust the bits that get mapped and executed on your system.

At the same time, the benefits to shared libraries are really diminishing: https://drewdevault.com/dynlib.

It's not just that it's easier to ship statically linked or bundled apps. It's more reliable and doing the alternative doesn't have many benefits.

And as an application developer, hoping that your users aren't exposed to danger by linking ancient openssl versions or whatever is also a problem.

Frankly, I think dynamic libraries are less secure (at least when talking about a desktop environment, not servers) because sometimes package managers update dynamic library dependencies without actually checking if binary compatibility was preserved in the new version, causing various crashes and bugs (I experienced this first hand with an archlinux package of a software I was developing).

That may true on Linux, but it is not the case on Apple platforms. On Darwin the base system is immutable and the dylibs embedded in an application bundle cannot be changed without invalidating the codesignature. In order to have this sort of issue occur you need to opt out of multiple security settings that are enabled by default (such as the hardened runtime and library validation) AND then be sloppy with your use of relative paths or dlopen() calls.

Yeah this is the right way of doing it. The archlinux (and most other distributions') way is profoundly stupid.

I think you totally missed what the issue is here. In that scenario, Apple could still patch a system lib and it can break your application.

It's not a question of the library updates being untrustworthy and code signing by the vendors fixes it. It's the library updates themselves breaking shit, not intentionally.

Static linking prevents that, at the cost of disk space and memory and missing out on updates that might not (usually won't) break your app.

Otoh, if you told me apple is more careful about breaking ABIs with updates to shared libraries, that is believable.

It's not a matter of "could". Apple DID this in the past.

In the mid-2000s, GCC switched C++ ABIs. Apple pushed out an update to Mac OS X 10.3 which replaced the static libstdc++ using the old ABI with a dynamic libstdc++ using the new ABI. This broke C++ compilation under Mac OS 10.3. They did not bother to update the compiler to use a new ABI; they had decided that if developers still wanted to compile for 10.3, their supported path -- compiling using the latest Xcode (available for 10.4 and up only) in its 10.3 backward compatibility mode was good enough. People still using 10.3 who wanted to build e.g., GNU software were just too niche a use case to warrant any kind of support.

Lesson learned: if you intend to build software on Apple systems, always have the latest shiny, or you risk being left in the lurch.

g++ ABIs changing in the 2000s was pretty painful on linux too.

But I think there is a fair point that arch linux probably doesn't care much about what breaks on its rolling updates to shared libs. I would guess you'd get less of that on, say, Debian.

>arch linux probably doesn't care much about what breaks on its rolling updates to shared libs.

This is a misconception. I'm not saying it never happens, archlinux is run by volunteers, resources are limited, but contrary to popular belief it isn't always the place that gets updates the fastest and when things break it's because upstream really screwed up and let something bad in that went unnoticed. If anything is noticed at all, arch will play the conservative card.

Bash is still on 5.1.x on archlinux because Bash 5.2 (that was released, by the way, almost a year ago!) introduced compatibility issues with older scripts that abused some bashisms (this makes a very solid case for only writing shell scripts in pure posix. Use shellcheck and checkbashisms.) Meanwhile almost every other distros jumped on 5.2 by now.

Arch will upgrade to the latest and greatest when changes are minor. We get the official upstream point releases that fix bugs and security issues fast. It does not mean Arch doesn't pay attention when it matters.

Most often when something break on archlinux it is because people aren't following the instructions from the arch-announce mailing list when there is a major transition, because arch is not a distro that automates processes. When grub broke for some users recently, it was because they didn't run grub-mkconfig, which should always be done when upgrading grub because grub has a very funky configuration file format (the config file you write is like a "source" for it to generate another, different config file that gives you NO guarantee about its format staying stable). Note that upgrading the grub package does not, in any case, ever upgrade grub itself, because archlinux is a distro that does not automate things for you. So people who had a broken grub did this to themselves, they ran grub-install but didn't run grub-mkconfig. Anyway, I would recommend systemd-boot to any user on modern EFI systems because it was written by much saner people.

Arch Linux's core philosophy isn't really about having the absolute latest packages. There were times when a Fedora came out with the latest Gnome before the latest version reached the stable repositories in Arch.

Arch's philosophy is to give you a Keep It Simple, Stupid system. There is no automation beyond what the software packaged provide. There is no splitting packages into many tinier packages like other distros that are very annoying with their -doc, -dev and so on packages (disk space is so cheap, why would you NOT want the documentation to be present??). You don't have to hunt for funky names of dev packages when you need to compile software, if you already have the dependencies installed, they also come with what you need to build against them. The arch packaging format is the simplest of all distro, with the exception of slackware which does not have dependency management. What few system tools exists to manage the distro are all written in shell scripts (mkinitcpio, arch-chroot, pacstrap, pacdiff, mkarchiso..) except for the package manager, pacman, being in C. You could say that being a rolling release is a side effect and not the main purpose. Since there is no automation and it is a simple system on an architectural level (not as in "user friendly"), adapting to gradual changes is much less painful than doing major releases every once in a while like debian stable because with the KISS philosophy it would compress a serious amount of work to do on the side of the user to handle so many transitions at once on their own.

Debian's care about whether things break or not, that you mention, depends on which edition of debian we're talking about. Debian stable cares a lot about breakages and upgrading from a stable to another is always a smooth process. They go to painstaking lengths to make it work.

Debian unstable on the other hand.. I've had times when they introduced new library versions that broke many other packages and I found myself unable to install software I wanted to try because the repositories were broken in that way. Debian unstable is not the "rolling" alternative some people think it is. If you want a rolling distro, use a rolling distro. Ever since I tried arch many years ago, I left unstable and never looked back. A library that breaks many packages that are part of the arch repositories will not leave testing. They can however break things that are from the AUR, there is no official support for the AUR pkgbuilds.

Not even the last time they did something that. Just in the past six months, I noticed I was no longer able to link against certain C++ libraries being installed via Homebrew, because some change in the ObjC runtime meant that whatever I was running was unable link libraries built with newer compiler versions (various ObjC symbols would be missing) - presumably the binaries installed by Homebrew had been built on a newer system - and the current Xcode tools wouldn't install on Monterey, you had to upgrade to Ventura. Meh.

> Static linking prevents that, at the cost of disk space and memory [...]

Some deduplication work for disk and memory pages should be able to help here?

(It might need compiler and linker support to produce binaries that are easier to deduplicate. Eg you might want to disable unused-function elimination for your static libraries and restrict whole program 'Link Time Optimization'? Or you might want your deduplicator to be a bit smarter and store diffs internally, like git does? Or your build system can work this way in the first place and produce diffs against common bases.

I don't know what's optimal or even practical, but you can do static linking and still save memory and disk space.)

In principle, yes? But it's a much more roundabout way of saving space. Reducing or avoiding optimizations like LTO or unused function elimination is at odds with minimizing binary sizes and maximizing performance. It's asking developers to prioritize the disk usage of the system as a whole over the performance of their own software.

You are right. But it's the same roundabout way that git is using.

Older version control systems like subversion used to store diffs.

Users of Git care a lot of about diffs between versions. And typically treat a specific commit as if it was a diff. Commands like 'git cherry-pick' re-inforce that notion.

However, internally each git commit is a complete snapshots of the state of the repository. Diffs are created on the fly as derived data.

Now even more internally, git does store snapshots as mostly as deltas. But to close the circle: those deltas have nothing to do with the diffs that users see.

This sounds very roundabout, but results in a simple and performant system, that doesn't ask the user to make compromises.

My suggestion for static libraries was along the same lines, if a bit clumsier.

I think deduping distinct but equal memory pages and marking them as copy on write is a relatively new feature in kernels. Easily 20 years later than shared libraries becoming common.

In this case you wouldn't need copy-on-write, because executable pages aren't writeable these days. https://en.wikipedia.org/wiki/W%5EX

I'd say from a kernel perspective they should be copy-on-write.

Firstly, the generalized kernel mechanism which scans for equal pages and de-dupes them [which by the way is disabled by default on Linux] probably doesn't care about if it's working on data or code; it seems like the primary use case at its introduction was for KVM, which, a kernel probably loads code pages and hence writes to them at some point, such as when it reads them from disk.

Second, someone can use mprotect(2) to make them writeable.

The GP specifically talked about an inadvertent dylib hijacking, which is prevented by the mechanisms I described. You are talking about a platform ABI break, which while unfortunate does occasionally happen due to significant technical issues (or sometimes by accident).

Apple does spend a significant amount of effort to avoid breaking supported ABIs. There have definitely been issues though, and especially early in Mac OS X while learning how to deal with upstream open source projects that don't care about ABI. In this specific case the result was Apple funding the development of libc++ and factoring it into libc++ and libc++abi specifically do prevent this sort of breakage in the future. Another example would be about a decade ago when Apple removed the ssl headers for the SDKs and told developers to either use SecureTransport or include their own SSL libraries, since depending on openssl's ABI was not feasible.

> The GP specifically talked about an inadvertent dylib hijacking

You're wrong.

> because sometimes package managers update dynamic library dependencies without actually checking if binary compatibility was preserved in the new version,

That is a very clear description of an ABI break.

They explained an issue they had on Linux, which was that he installed a package with a library that broke ABI with a client. I explained how it is rarely an issue on macOS because the risks of it are limited by how the system is constructed (immutable base system, so no mix and match package issues) and how linking policy is configured (by default binaries can only link to the embedded dylibs referenced in their bundle's code signature or platform binaries). So yes, I phrased my response in the context of a specific subset of the issue (running code from the wrong library) because all of the rest of the ways that happened are (modulo bugs) prevented by construction.

Package managers do not modify the base system on macOS, and the binaries installed by them will not have code signatures will not be trusted by executables built with the default ecosystem's policy, so they cannot impact binaries outside of their control unless those binaries have been specifically opted into a reduced runtime mode, which makes the change of an ABI break way lower (but not 0) than on Linux... I don't even understand what is controversial about that statement, the systems are built with different goals and engineering trade offs.

And before you tell me about how some system upgrade broke homebrew... of course it did. Many (most?) homebrew packages are opted into policies much closer to the behavior on Linux (flat namespaces, undefined dynamic_lookup, opted out of hardened runtime) because they depend on Linux like semantics due to being multi-platform. That also means they get none of the protections afforded by the system to prevent these issues. If they adopted two level namespaces, @rpath, limiting usage to APIs available in public SDK headers, and using min deployment targets they would be fare more resilient to system upgrades, but that would also entail an ongoing maintenance burden for packages primarily developed on and for Linux.

IOW, if your point is dynamic linking provides primitives to build fragile systems, then sure, I agree. If your point is all dynamically linked systems are significantly more fragile than statically linked systems I disagree (though I can actually point out cases where ABI mismatches have occurred in both static and dynamic binaries). If your point is that a system that allows fragile behaviors to be opted into by power users is inherently more fragile for normal users, then I also disagree (though I concede it may be more fragile for those power users).

A package update to an important shared library in a Linux distro is the equivalent to a system update on the Mac, if I read you right you are correct in saying that in a long-winded way.

I feel like you may have some Mac fetishism going on that is leading you to see those two as more distinct than they are. The topic at hand is an ABI break after a dynamic library gets updated legitimately by its vendor. Code signing and "injection" is tangential.

It is not fetishism to point out something behaves differently than Linux, especially in story about a technology introduced on Apple platforms.

Technologies don't exist in a vacuum. This thread pointed out a problem with dynamic libraries that cannot occur on Apple platforms because dynamic linking does not exist in a void but is part of an ecosystem that exists beyond the dynamic linker. The fact that in that context you keep insisting on ignoring any technology not present on Linux is sort of baffling.

Incidentally, the same strong security measures have to be disabled to change the code templates that start every new Swift code file with a comment block containing the current date and the name of the person that created the file. I will never understand what led to this— it’s as if they created this before source control existed.

It’s ‘underdocumented’, but you don’t have to do that to remove those.

If you search for “Customize the header used for new files” in Xcode’s help, you can find:

Change the text used for the header of a new file by setting the value of the FILEHEADER text macro.

Most of the templates start with

  //___FILEHEADER___

, so you can’t remove the initial “//“, but you can remove the rest.

Changing the value of the macro can be done by creating or editing a .plist file:

Text macros are symbols that are expanded in-place to the specified text. They are used in files and other places in Xcode, such as the header text for a new file or the project name. You can customize existing macros and add your own macros in a project, for all users of a project, or for all of Xcode. Customizing a macro requires two things:

- A plist named IDETemplateMacros.plist at an appropriate location.

- An entry in IDETemplateMacros.plist for the text macro.

Xcode looks for the value of a text macro in the following locations and uses the first matching macro:

Project user data: <ProjectName>.xcodeproj/xcuserdata/[username].xcuserdatad/IDETemplateMacros.plist.

Project shared data: <ProjectName>.xcodeproj/xcshareddata/IDETemplateMacros.plist

Workspace user data: <WorkspaceName>.xcworkspace/xcuserdata/[username].xcuserdatad/IDETemplateMacros.plist.

Workspace shared data: <WorkspaceName>.xcworkspace/xcshareddata/IDETemplateMacros.plist.

User Xcode data: ~/Library/Developer/Xcode/UserData/IDETemplateMacros.plist.

This is a 40+ year old problem that's been solved but not widely solved.

There are no security implications here because it's strictly a configuration management problem. If only root can do software updates, then there is no security problem. Don't modify software with incompatible versions, and there is also no problem.

Package/dependency management on single /usr/lib Linux distros is a disaster because it doesn't allow for side-by-side installations of various versions leading to upgrade dependency hell. Nix solves this by isolating and versioning dependencies. GNU stow and habitat also address this.

That's why distributions like Debian/Ubuntu/RHEL exist. They provide a stable toolchain to build on top of _within_ a major release of the distro.

Documentation at

https://developer.apple.com/documentation/xcode/configuring-...

They better keep the new linker open source like they did for ld64 (despite delays), or it is going to become a private ABI nightmare.

> That does mean if you rely on bundle lookup support, you should update your minimum deployment version to iOS 12 or later to use mergeable libraries. But if you don't rely on these APIs, you can disable this support with the new linker option -no_merged_libraries_hook. Then you won't need to update your app's deployment version.

I guess no one told the linker team that Xcode 15 was going to bump the minimum deployment target to iOS 12? Sort of awkward that no one caught that this bit of the talk isn't actually applicable to anyone.

Hardly anybody supports iOS 11 these days anyway. Every device capable of running iOS 11 can be upgraded to iOS 12, which was released almost five years ago.

Weird. Yes, link time is a big issue at development time, but with the new development of ld-prime and sold, seems trading that with dylib is not a big of a deal.

On the other hand, the other big use of dylib I saw for apps to embed (other than providing 3rd-party libraries for other people to use), is to share code between main app binary and extensions, which mergeable library doesn't seem to support.

I guess the biggest use is indeed to deliver 3rd-party close-source libraries?

Many, perhaps most?, dylibs that a Mac app ships with are all within the app bundle itself, and signed as such, so dylib sharing is not a common use case.

Yeah, and the only reason they don't static link is because these dylibs are some 3rd-party closed-source libraries to avoid a lot of version clashes comparing to ship static lib naively.

Otherwise you put some code in dylib so your app extension can share some code with the main app, but that is tricky due to different RAM usage restrictions between app extension v.s. main app.

RAM usage restrictions?

For example: https://tailscale.com/blog/go-linker/ talks about how they slim down the Go runtime to workaround the 15MiB memory limit for network extension.

But what does that have to do with dylibs?

If you have RAM usage restrictions, aren't they for the whole process, so the main binary and all its dylibs?

Yeah, I am not explaining myself very well. I meant: people try to sharing code between main app and extensions with dylib. If they don't do that, they have code-duplication in the app bundle so the binary size is main code + 2 * shared code + extension code. However, it becomes more complicated because if you just naively sharing the code, your sharing part will be bloated, therefore, making the RAM usage higher than a static linked one (because static linked sharing code can do dead-code elimination better).

Will this make it possible to link in standalone libraries as like a single bundle easily and distribute that without having to worry about local installations?

E.g., I tried to bundle ffmpeg as a library with my voice memo transcription MacOS app^0, but it was too difficult, so I just went with building a standalone ffmpeg binary (~40MB), and instrumenting it with a bash script that I called from the Objective-C code, ha ha ha! :)

0: https://apps.apple.com/us/app/wisprnote/id1671480366?mt=12

Just a side note, what you're doing sounds like a violation of the FFmpeg license. https://www.ffmpeg.org/legal.html

What exactly are they violating? Isn't running a binary with a bash script ok?

Just based on what the comment says, they're distributing a compiled FFmpeg, presumably not with source or attribution. I can't check to see if there's information in the app but there's no mention of it on the store page or anywhere else I can find either.

They could be fine, but going through the checklist FFmpeg provides for legal considerations (not legal advice), they seem to be doing the opposite of all of them.

I haven't checked their specific app ($119.99) but it's common for packages to have OSS attribution and copyright notices as a dialog or something that the user can click on to see. Since they're presumably not modifying ffmpeg, there's also no source that needs to be provided.

For example, my Chrysler car infotainment has an option in the system settings to see all of the copyright notices and OSS info that goes into the system.

> If you’re using AGPL-licensed software like a database engine or my own AGPL-licensed works, and you haven’t made any changes to the source code, all you have to do is provide a link to the upstream source code somewhere, and if users ask for it, direct them there. If you have modified the software, you simply have to publish your modifications. The easiest way to do this is to send it as a patch upstream, but you could use something as simple as providing a tarball to your users.

(AGPL is generally stricter than LGPL)

https://drewdevault.com/2020/07/27/Anti-AGPL-propaganda.html

GP was saying that statically linking would have been a violation. That is what GGP was trying to do, they fell back on running an executable because they didn't manage it. They're asking how to do it, but that wouldn't be legal (unless you offer a way for people to re-static-link with a modified version of ffmpeg, which is not easy if you don't want to publish your sources).

Even distributing the standalone ffmpeg executable might be a violation, if there have been changes to the ffmpeg code and it's closed source.

If there have been changes to ffmpeg code, then all that's needed is making it possible for the user to obtain the changed ffmpeg code. Voila. There's way too much FUD.

I said "might" and "if"... where's the F, U, or D? If the source is published then that avenue is fine.

Yeah, I was wrong and per part 1 of https://www.gnu.org/licenses/gpl-faq.html#LGPLStaticVsDynami... this ios developer actually needs to provide the source for that app if they statically link. Otherwise it is an LGPL violation

Ha ha ha! :) Thanks for the side note, do you have any answer to the actual question? Ha ha ha! :)

Really happy to see this and the child in me hopes to see something like this get "up streamed" into the ELF spec.

This looks really hype-y. AFAICT a "merged" binary is just a statically linked one. The only problem it solves is a self-inflicted one - failure of the old static linker to prune or de-duplicate stuff in linked libraries (particularly ObjC-specific stuff). It notably does not solve the main problem with static linking - old insecure code which would have been fixed with an updated dynamic library but is instead "stranded" by having already been copied into executables (or bundles and no reasonable person would quibble about that) never to change again. Also: provenance, supply chain, etc.

Yes, I'm aware that superkuh beat me to this last point, and also that updating dynamic libraries can also cause breakage. But I still think it's important to note that this isn't really advancing the state of the art like Apple would like you to believe. It's just a new middle-of-the-road approach with its own possibly-positive tradeoffs and pitfalls. Nothing here that wasn't already considered at least thirty years ago, and whether they were right or wrong to choose another path is less relevant than the fact that it's not new.

The problem being tackled here is link time in debug builds. This affects all platforms.

> The problem being tackled here is link time in debug builds. This affects all platforms.

I've worked on many projects, big and small, and the link time of debug builds was never a problem that was worth fixing.

In fact, even today I was discussing with a coworker the inefficiencies of the project's build process, we literally commented that having to only link a huge subproject is a major blessing.

For Google at least, link times are sufficiently important that the company has rewritten the linker from scratch twice--both open source. The Gnu-Gold linker which ships as part of gnu binutils and subsequently the ELF version of llvm's lld.

So although you might not encounter issues with link times (debug or otherwise), it is a multi-million dollar problem for big companies like Google and Apple. Both in resources and engineer time.

> So although you might not encounter issues with link times (debug or otherwise), it is a multi-million dollar problem for big companies like Google and Apple. Both in resources and engineer time.

I appreciate your appeal to authority, but I worked at a FANG on a project that had over 20 GB worth of static and dynamic/shared libraries.

Linking was never an issue.

Err, "Google has rewritten the linker twice. Both times with the stated goal to make link times much faster." isn't an appeal to authority. It's evidence that the company has found speeding up linking to be worth millions of dollars. Otherwise it wouldn't have done it.

They surely weren't doing it for fun.

> Err, "Google has rewritten the linker twice. Both times with the stated goal to make link times much faster." isn't an appeal to authority.

It is, and a very weak one considering Google has a history of getting people to work on promotion-oriented projects.

https://news.ycombinator.com/item?id=31261488

> It's evidence that the company has found speeding up linking to be worth millions of dollars.

It really isn't. You're buying into the fallacy that a FANG can never do wrong and all their developers are infallible and walk on water. All you're able to say is google did this and google did that, and you're telling that to a guy who has first-hand experience on how this and that is actually made. You never mentioned any technical aspect or more importantly performance differences. You only name-dropped Google, and to a guy who already worked at a FANG.

Linking was never an issue.

There are many FAANG customers who care about link time; some of them are also FAANGs, but certainly not all. You're falling into the libertarian trap of thinking that because it didn't happen in your experience, it could not possibly happen to anyone.

Want to throw out the technical aspects and performance differences. Judge for yourself:

On initial contribution to the gnu binutils project, the Google developer for Gold claimed a 5x speed improvement: https://sourceware.org/pipermail/binutils/2008-March/055660....

In a public talk on llvm lld, the Google developer for LLD claimed another 5x speed improvement:

https://llvm.org/devmtg/2017-10/slides/Ueyama-lld.pdf

If you had 20GB of dynamic and static libraries as a part of your build the only thing that can be a bottleneck is linking

> But I still think it's important to note that this isn't really advancing the state of the art like Apple would like you to believe. It's just a new middle-of-the-road approach with its own possibly-positive tradeoffs and pitfalls.

It also seems that this new library format barely solves any problem and in the process bumps up the number of library types that developers need to onboard and possibly support.

> "It also seems that this new library format ... bumps up the number of library types that developers need to onboard and possibly support."

No, a mergeable library is just a special type of dynamic library, which adds some additional features and metadata. I can't imagine why you'd distribute both mergeable and non-mergeable dylibs?

In fact, rather than having to support more library types, developers may actually have to support less - because it may eliminate the need to ship both a static lib and a .dylib with your framework?

There is a bit more to it than that. Yes, it was always possible to use a mess of build rules and shell scripts to make your debug and release builds swap between fully static and dynamic libraries, but it was a lot of work, and was difficult to maintain. The novelty of mergeable dylibs is that they now make it trivial to switch between the two without all of that work. In particular it solves two large problems people tended to run into:

1. Static archives and dynamic libraries have different semantics with respect to symbol lookup. In particular, due to two level namespaces multiple dylibs can exports the same symbol without it being a runtime collision since the binary importing them stores the library a symbol came from in addition to the name. This is different from static archives where you have sets of symbols brought in by .o file. That means it is often non-trivial to switch between dynamic libraries and static archives. Mergeable libraries solve this by allowing you to use the semantics of dynamic libraries and two level namespaces for content that will be statically linked.

2. Most people use frameworks, not raw dylibs. They do that for a lot of reasons, but the biggest one is to allow them to distribute non-code resources that are associated with the library. This is a common problem that has been solved in various ways (Windows embeds the resources in the DLL files, classic Mac OS depended on resource forks, etc). Mergeable dylibs are completely supported by the runtime in such a way that enough of the dylib's identity is preserved so that things like NSBundle continue to work as a way to find the bundles resources despite the code itself being merged.

I'm not seeing any hype. They have shipped a nice new feature for their build system. I don't see any claims of it advancing the state of the art, or life-changing, just that it improves certain work flows in an easy way, and provide a convenient way to use it for their developers.

This sounds very much like... prelink?

Not quite, but it's closer to statifier https://statifier.sourceforge.net/ or ermine https://www.magicermine.com/

You can definitely do most of the same process in hacky ways on Linux already. I don't think you can use it for making the merged shared lib to use for compilation later... But it's only a few hacks away.

From the video:

> Mergeable metadata roughly doubles the size of the dylib.

I think they just bundle the static lib and dynamic lib together. I think there is a wasted opportunity to do something much smarter here, but maybe that will follow.

We already compile static libraries with PIC, and use the same object files to compile dynamic libraries. In theory the two could be bundled in a way that shares most of the data, as the machine code is literally the same. They are both linked from the same object files.

Reminds me of some recent work in nixpkgs where the idea is to get a mostly-static builds that only depend on libsystem on MacOS. https://github.com/NixOS/nixpkgs/issues/214611

At some point the difference does seem to become mostly "what do you do during linking".

give me static αpε or give me death!

It won't make much difference to me, until it's supported in SPM (which will probably be soon).

Reply on news.ycombinator.com