ThinLTO: Scalable and Incremental Link-Time Optimization for LLVM | Hacker News Reader

ThinLTO: Scalable and Incremental Link-Time Optimization for LLVM

blog.llvm.org

99 pointsDannyBee18 comments

There's a session from WWDC 2016 that talks about this - https://developer.apple.com/videos/play/wwdc2016/405/

The LTO part runs from 14:00 to 22:00. The streaming seems to only work in Safari (for Macs), but you can download the mp4s in HD (1.2 GB) or SD (300 MB).

They do a good job of describing how it works and why this is kind of a big deal.

Interesting.

Google built it because it's pretty necessary to scale to the size application we build.

It looks like Apple started using it just because it's faster anyway ;)

For non-Safari-users, the download is hiding in the grayed-out "Resources" tab.

The link to the PDF about the Incremental LTO is here, thanks to aseipp:

https://news.ycombinator.com/item?id=11962571

Relevant Rust discussion: https://www.reddit.com/r/rust/comments/4p4l5k/

I didn't understand fully what it took to achieve this but simply looking at the numbers it is so clear how impressive the work is. Near or better performance with enormously fast build time, significant improvement in incremental compilation and enormous reduction in memory usage. Its unusual to see an optimization attempt succeed from so many different angle.

It's been in the works for quite a while (I remember reading some of the initial RFC postings), although I haven't seen people talk much of it until now. To be fair, I believe this work was primarily done by Google, who also have contributed a lot code to LTO/IPO-based systems in GCC before (such as "Lightweight IPO" or LIPO in GCC) as well as obvious, ongoing Clang/LLVM work. So I imagine they probably had a very good idea of what was necessary to achieve all that... But yes, the results are really impressive!

Here are some slides from EuroLLVM last year with some pretty graphs and comparisons to traditional LTO/LIPO: http://llvm.org/devmtg/2015-04/slides/ThinLTO_EuroLLVM2015.p...

"Its unusual to see an optimization attempt succeed from so many different angle."

David, Teresa, and Mehdi are really amazing engineers.

(also, this is the third or so LTO system some of them have built :P)

That's all well and good, but where can I get the code? Where can I play with this?

How can Apple be shipping it (?) when LLVM is an open source project and it's not in the SVN head yet?

Is there a special branch you need to use? A different repository perhaps?

>How can Apple be shipping it (?) when LLVM is an open source project and it's not in the SVN head yet?

Apple ships their own proprietary forked version of Clang+LLVM, this is for example how they implemented Swift, which was made open source long after it was introduced as part of Apple's proprietary toolchain.

It's in TRUNK in with the rest of the llvm-lto stuff.

Is LLVM still using SVN??

Is it compatible with ccache?

Is all this functionality encoded within .o files or does it create a separate cache file or directory?

IIUC, it's creating .bc (LLVM bitcode) files as input for the final link stage.

Thanks. And I learned a new acronym. Oh that crazy internet.

Nice work, mainly because the linker seems to be the most mysterious aspect of computer architecture.

Reply on news.ycombinator.com