And what dependency management or package system would you use to install that library? Distributing the source, especially for self contained libraries, is easily managed with the rest of your source code.
LTO is awesome, as some code does benefit from optimizing for a minute to get 5% more speed for the next week (and sometimes even more aggressive parameters for the optimizer, if you run the code for long enough to matter).