Hacker News Reader

Comment by gjvc | Hacker News Reader

While automated, it still makes CI/CD take quite a long time

See about using ccache -- https://ccache.dev/

I wouldn't recommend ccache (or sccache) in CI unless you really need it. They are not 100% reliable, and any time you save from caching will be more than lost debugging the weird failures you get when they go wrong.

please provide evidence for this assertion.

Why are you so skeptical? Think about how it works and then you'll understand that cache invalidation bugs are completely inevitable. Hell, cache invalidation is notoriously difficult to get right even when you aren't building it on top of a complex tool that was never designed for aggressive caching.

Just search the bugs for "hash":

https://github.com/ccache/ccache/issues?q=is%3Aissue+hash+is...

You can't cache based on the file contents alone. You will also need to cache based on all OS/compiler queries/variables/settings that the preprocessor depends on, since the header files might generate completely different content based on what ifdef gets triggered.

And that’s not impossible, just tedious. One tricky (and often unimportant) part is negative dependencies—when the build depends on the fact that a header or library cannot be found in a particular directory on a search path (which happens all the time, if you think about it). As far as I know, no compilers will cooperate with you on this, so build systems that try to get this right have to trace the compiler’s system calls to be sure (Tup does something like this) or completely control and hash absolutely everything that the compiler could possibly see (Nix and IIUC Bazel).

In C++ the __has_include preprocessor expression has been standardized since C++17, I'm not certain if C has standardized it yet though.

It’s not about that, that’s not relevant to ccache at all. (And yes, C23 does have __has_include, though not a lot of compilers have C23 yet.) It’s about having potentially conflicting headers in the source file’s directory, in your -I directories, and in your /usr/include directories.

Suppose a previous compile correctly resolved <libfoo.h> to /usr/include/libfoo.h, and that file remains unchanged, but since that time you’ve installed a private build of libfoo such that a new compile would instead resolve that to ~/.local/include/libfoo.h. What you want is to record not just that your compile opened /usr/include/libfoo.h (“positive dependencies” you get with -MD et al.), but that it tried $GITHOME/include/libfoo.h, ~/.local/include/libfoo.h, etc. before that and failed (“negative dependencies”), so that if any of those appear later you can force a recompile.

Oh yeah that can cause lots of weird problems. I've run into that sort of issue a lot when cross-compiling, cause often then you might have a system copy of a library and a different version for the target, that can be a real pain.

Maybe run every build version in their own container?

please read the documentation before dispensing uninformed advice like this -- it works using the output of the preprocessor and optionally, file paths

Here’s an example of a common Rust dependency which had a very subtle caching issue: https://github.com/dtolnay/thiserror/pull/279

Reply on news.ycombinator.com