Compile your software

So much advice in software development presupposes that everyone has the same single goal and at least one inexhaustible resource. In this case, time:

- To read and understand all relevant parts of the build pipelines for each performance-critical part of the infrastructure. In a relatively simple web application that could mean at least a web server, a caching framework, a database and a message bus.

- To debug any build failures, which could be plentiful and hard to parse until you're very familiar with the build infrastructure for that particular piece of software.

- To benchmark the build outputs in comparable ways with realistic configuration and inputs.

- To repeat the above whenever any part of the stack changes significantly.

It also could introduce bugs given your company might be the only one earth with that combination of compiler, kernel and libraries.

Only if you do it for everything. But it might be worth it for the most performance critical components.

Author here.

> The argument here I suppose is live on the bleeding edge? Or swap compilers/allocators as necessary? Or use O3? I'm not entirely sure.

My point is that when you are using a software, especially open source, you should evaluate it, and understand it. BTW, we are running arch in production, with our own repos.

> It is PITA is keeping up with compiler changes and library changes of third party software. In this case, throwing in different malloc implementations in there too, not to mention different libc implementations. With all those variables for each component you deploy you're probably less likely to understand what's going in your software.

in my example, the libc implementation is always the same! it's just that it's outdated on all main docker images. Furthermore, Redis already use a different malloc implementation (jemalloc), but in the makefile, they support also the standard malloc and tcmalloc, so throwing in another one is very easy.

> Maybe for a small app with 2-3 extra pieces or something that needs to really be optimized this is good advice, but it sounds like a lot of work for significant footprints. Or when you need security/performance. of course, if you have 2 servers, it's useless, but we have more than 4000 servers running in production on arch, so it's worth it for us.

> It would be nice if the official docker images had some tags that were better optimized, within reason. or just be up to date

This is one thing where I see Rust (and probably Zig) making headway. A lot of the newer Rust software isn't in package repos yet, but I don't mind doing a Cargo build. It might take a while to compile, but it always seems to just work.

> The sad fact is that a lot of software is difficult to compile; isn't documented well, something that is worse for building; won't work well if installed in a non-standard way, whether that is final location, different supporting libs, or different platform; and can take a long time.

So are you ready to deploy in prod a software that is so hard to compile?

> I'm happy nowadays when I see there's a binary available, no mucking around with gcc/clang/llvm - just trying to work out which one, let alone which version! - no diving down a rabbit hole of compiling dependencies that then need other dependencies compiled… no deciphering Makefiles that were written in a way that only a C guru can grok, with no comments.

But that's my job, as SRE/DevOps/whatever new fancy name!

> Whatever the benefits are, I prefer sanity.

Sanity of having a very old software, with backported features that are only on this distrib? I prefer to trust the engineers from the software that I deploy.

Try pkgsrc.

That doesn't match my experience. Two decades ago everything was in constant flux and more unreliable, but nowadays it's rare to find broken builds. It's only difficult if you distribution puts headers and such in separate packages, or splits up stuff so much that it's hard to tell what you actually need. Can't remember the last time I had to intervene to compile a random GitHub project I wanted to try out. It just works.

Agreed. I thought this was an interesting post but was baffled by every y axis.

And better titles. "Performance" is confusing. I think he means "Performance of different base images for Redis"?

Thanks for the feedback. Let me find a better way to make it more clear!

Read the article. I compiled redis with Zig, O3 native, and mimalloc!

-march=native will still give you better performance. It's not just about the instruction set, but also heuristics taking into account cache size, latencies, topology and other things. Intel for example has this quirk that aligning functions (and other jump targets) at 32 byte boundaries speeds up funct8 calls and jumps. I haven't tested it but I suspect you'd gain more from -mtune=native with the generic x86_64 target than -march=native. Some loops that can be autovectorized with AVX instructions will probably be faster though. But cache size especially is important for deciding if some optimization is beneficial or just leads to stalls due to thrashing.

that's true, but without some crazy guy like me to check it, you would not know! :-)

Nice one, I never heard it!