Hacker News Reader

Comment by DCKing | Hacker News Reader

Not a macOS dev but maybe one can clarify: is the title "No more system shared libraries in macOS 11" actually true? The documentation reads to me as a restriction on how you read dynamic libraries (is using only dlopen controversial here? also just trying to understand), not on which system libraries are actually there. It seems dlopen with custom system dylibs would still work if there is also a means to get your dylibs in the system cache? Based on that piece of docs alone I see no reason to grab my pitchfork just yet.

Anyway, I was expecting a few shitstorms to start happening over macOS 11. Apple not announcing any major deprecations (edit: or removals) for this transition - even OpenGL/CL is still there, depecrated - was suspicious.

View on HN

You're correct. For 3rd-party provided dylibs, it also isn't necessary to get them into the system cache.

dlopen() will look for the dylib first (as today), and if the file doesn't exist, it'll look in the shared cache.

So the only people this actually impacts are those who stat() system-provided dylibs before calling dlopen(). They should just skip the stat(), and go straight to calling dlopen().

Certainly not worthy of the end-of-the-world vibes in the twitter thread...

Thanks for the clarification. I'm guessing people are just looking for the inevitable pitchfork worthy big bad breaking change, but we haven't found one yet (although I'm slightly annoyed my Macbook will be out of official support with this release).

@dang - I flagged this for the editorialized title, in my view it's worth updating this to something like "Restrictions on reading dynamic libraries in macOS 11".

> They should just skip the stat(), and go straight to calling dlopen().

But `stat`ing it used to be fine on a Mac. And `stat`ing it is fine on other OSs. Why force software to change for no apparent reason just to appease the arbitrary whims of Apple? (And if it's not an arbitrary whim, you'd think they'd at least hint at the upsides of this, no?)

Even ignoring the race condition, has stating shared libraries ever been documented as a method to ascertain existence of a shared library?

My guess as to the reason: I would guess the new way is faster, more secure, or takes less memory. Leaving an unused copy of binaries around would be wasting disk space (and yes, disks are enormous nowadays, but SSDs aren’t)

It may cache data from Apple’s servers, but calling it a cache sounds incorrect, though.

I wouldn't think it's any of those reasons. I'd rather think it's to transparently serve up different libraries based on the caller, even if the name is the same, in the name of compatibility or something like it.

>Why force software to change for no apparent reason just to appease the arbitrary whims of Apple

Why assume there's no reason?

To be fair, the reason is certainly not apparent.

The reason is immediately apparent. They're eliminating a redundant copy of the libraries to save disk space.

That's definitely not the reason, given that symlinks are still a thing.

You can't symlink into a byte range of a file. All of the system dylibs are stored in a single file.

Also, the dyld shared cache isn’t just some archive containing exact copies of the dylib files. Among other things, the cache builder pre-resolves references between different dylibs in the cache. This saves time on process startup, and also allows pages to be shared between multiple processes even if they contain relocations, saving memory. But it means that even if you could symlink to a byte range, you couldn’t just have the original dylib paths be symlinks to parts of the cache. (It might be possible to make this work with enough effort, but it would be very nontrivial.)

Note that the dyld shared cache itself has existed for a long time; the only change here is removing the original copy of the dylibs to save disk space.

To save disk space and time during os upgrades. Specifically, the last 2 times my old, slow Mac upgraded its OS, it spent many many minutes rebuilding the dyld shared cache (during which my Mac was unusable of course).

That too.

Stat then dlopen is a toctou (time of check/time of use) bug. There's a race condition between the check succeeding and the dlopen being called in which that library might be replaced by a malicious one.

Like all race condition bugs it is probabilistic, but it can be won. By contrast dlopen either succeeds or does not, and once you have the handle to the library you are good.

Actually windows also works this way, and has done since wow I think vista if not XP? There's a knowndlls registry key. Large parts of the windows api are contained in a few DLLs; these are loaded on system startup. When resolving shared libraries if one of these names is requested the normal DLL loading procedure of checking paths isn't even followed - it just links in the already loaded DLL from the cache.

It actually wasn't designed as a security feature, but for speed. No matter what the OS does there's little need to continually load these DLLs from disk when we know they are already loaded to even display the desktop.

Of course on windows you can still do the windows equivalent of stat'ing these DLLs but nobody ever does.

> It actually wasn't designed as a security feature, but for speed. No matter what the OS does there's little need to continually load these DLLs from disk when we know they are already loaded to even display the desktop.

FWIW, this optimization is unnecessary on modern Unix systems with a unified buffer cache because libraries loaded from disk are already CoW'd into the user process, without the need for special semantics. It's a pertinent point, nonetheless, but I'm guessing Apple made the switch in macOS for other reasons, perhaps related to disk space, its less than ideal ASLR scheme for Mach-O, or something related to code signing and sandboxing.

I second the TOCTOU concern. It's a bad code smell and a poor habit to stat and then open a file, period. (Likewise for the more general pattern.) There are reasons to do it (some better than others), but they're exceptional. You should never be eager to do it, and when you do you should rule out all possible security exploits or bugs, which is usually more work than abstaining from the superfluous hack. Breaking such code patterns is worth the cost in backward compatibility, IMO, for all the exploits and usability headaches this coding pattern leads to.

Classic linking uses a path to a library at link time. I suppose Apple may have patched ld to work around this choice, but it still begs the question — why change this? Who was it hurting?

Not in every OS or programming language, it doesn't.

Linkers just need to somehow find the binary dependencies across the link path/staging.

> why change this? Who was it hurting?

Performance.

And security. If you sign a single cache across millions of systems you know that that is the one true non-tampered cache.

Can you elaborate beyond a single word? It is certainly not immediately obvious to me how this benefits performance, or I wouldn't have asked.

Here's one optimization: http://sealiesoftware.com/blog/archive/2009/09/01/objc_expla...

http://www.cydiasubstrate.com/id/727f62ed-69d3-4956-86b2-bc0...

The Apple linker has used `tbd` files, a textual description of a dynamic library's interface, rather than actual dynamic libraries at link time for iOS for several years now. Prior to that it was using stub dylibs, which were essentially symbol-table-only dylib files.

https://developer.apple.com/documentation/macos-release-note... has the text. dlopen will just work, but you can’t crawl through system directories anymore, hoping to find a shared library.

It wouldn’t surprise me if there’s no way to get your dynamic library there anymore, as this explicitly says it’s of the system provided ones.

OpenGL was deprecated last year

Everyone thought it was deprecated last year so that they didn't have to support it on ARM Macs, but it appears (not 100% confirmed until people actually get these things) like it is still there and working on ARM Macs (albeit deprecated)

It is officially documented as being available in ARM Macs.

Yeah but prerelease documentation has been wrong before.

Isn't that why it's called "Pre" release? If there is a chance of changes between the release and what comes before that I'd say that makes sense.

Say a 'release' turns out to have a discrepancy between documentation and reality, that's when the problems start to mount.

Reply on news.ycombinator.com