Shrinking a shared library
serge-sans-paille.github.io
serge-sans-paille.github.io
* -fvisibility=internal (stronger than hidden and must be used with a LOT OF CARE, e.g. never pass a function pointer to a function not marked explicitly with hidden or default visibility in that case as its ABI may change)
* -Bsymbolic / -Bsymbolic-functions / -fno-semantic-interposition (interesting explanations: http://maskray.me/blog/2021-05-09-fno-semantic-interposition)
* -fno-stack-protector (not recommended for libz aha)
* -fno-plt / -fno-ident (very small effect, more for the sake of completeness)
* -fvirtual-function-elimination (surprisingly not done by default, here I can sometime get back a few percent through this on OO heavy codebases)
* -ffunction-sections -fdata-sections -Wl,--gc-sections (largest improvement in my experience, sometimes this literally halved binary size for me when coupled with lto)
* -Wl,--as-needed
* -Wl,--icf=all (I heard horror stories about this but it works fine here)
It works by keeping an ar archive with PIC objects that would normally go into libc.so.6, determining all symbols used by all programs in the Debian installer, then mapping these symbols to objects, and linking a new libc.so.6 containing only the needed objects.
What's the point of making the so smaller to begin with?
99.9% of users will never touch all this, in return they get faster downloads of the browser and faster startup.
Yes, I suppose for pre release versions the calculation is different, but for a stable release there 'shouldn't' be any bugs.
You need some of that debug information to send backtraces back to the developers.
IDK on linux but on windows opening a 600mb exe (e.g. release with debug info) and a 15mb one (same but stripped) have a VERY OBVIOUS startup time difference. I assume the OS is scanning the whole binary through some antivirus for instance, and I'd assume the same to be true for DLLs used by the app, so all these kernel loading pages optimizations go through the trash.
Even Windows wouldn't do something so silly.
The obvious one is a smaller .so may have fewer pages of interest. If each function you call takes half a page, and you call 1/4th of the functions, chances are good you need to load more than 1/4the pages, because sometimes you'll call both functions in a page you load and sometimes you won't. In other words: if the .so is smaller, you get better packing.
The other explanation is prefetching. Lets say all functions take a full page, so packing into a page doesn't matter. When you call a function, which demand pages, chances are the kernel is also going to load adjacent pages; if all the functions you call are adjacent, prefetching saves you significant i/o time. Otoh, if your functions are sparse in the backing file, you will have to wait for i/o; which may be (marginally) delayed because of prefetching on earlier loads.
Users won't touch it directly, but I deem it a given that Chrome and its derivatives upload crash logs to the mothership by default, and IIRC Firefox also offers that as an opt-in. Same for Android (Firebase Crashlytics) and iOS, and I think there's also ways to hook yourself into JS frontend code to transmit uncaught exceptions to a backend server.
Although, thinking about it a bit more, are we sneaking up on the “why not use a static library” zone?
You can’t debug it anymore and you’ve possibly broken something because of the random functions you’ve pruned but you won’t be able to tell what you’ve broken. All to save literally a few bytes.
It is not used by a bunch of other applications besides Firefox so there isn’t much saving from being shared.
In addition, there isn’t a scenario where libxul would be released independently of Firefox for a security issue, so there is no advantage there.
You lose a lot of optimization opportunities with shared libraries and increase complexity.
> Why is libxul a shared library?
A lot of this is historical and is now very difficult to change without negatively impacting users.
> It is not used by a bunch of other applications besides Firefox so there isn’t much saving from being shared.
In the 2000s it was designed to do so, via XULRunner. You're also forgetting about Thunderbird and SeaMonkey (though I don't think that either of those can actually share libxul.so with Firefox anymore, but I digress).
But as to why it is the case now, there are still compatibility considerations that fall out from the era when Gecko+XUL were written as a runtime environment for hosting applications.
For example, I recall that during the Electrolysis project, we wanted to get rid of the plugin-container executable on MacOS and just host all Firefox child processes using firefox. Some "security" software would break Firefox because it made certain assumptions about the way that various Firefox binaries interacted with each other.
FWIW, Gecko used to consist of a lot more separate libraries, but over the years we recognized that there were a lot of performance issues that we could mitigate by combining them, so most shared libraries were folded into libxul.
And yes you could absolutely do a Matroshka with it: a VB6 app embedding Internet Explorer and the site embedding an ActiveX element written in VB6. Or inline Java applets.
Funny times... an era in which security was barely a concern. Instead, you had open-ish, usually well documented interfaces for everything.
What killed the idea though until Electron came along was simple: Browsers didn't even come close to feature parity. You had to write your own API layer to communicate between the website and the host application for everything going beyond XHR calls and file selector uploads. Electron took off once Chrome got enough native bridges that you could actually write capable web apps.