Hacker News Reader

Comment by superbatfish | Hacker News Reader

FWIW, I highly recommend using conda as a package manager for your own pure C++ packages. You can compile everything with the compilers they ship, and the only external dependency from your own system will be glibc.

Yes, it means you're using your own "mini-me environment", but you can share that environment across all of your C++ projects. As long as you build with:

LDFLAGS="-Wl,-rpath,${PREFIX}/lib -L${PREFIX}/lib"

... then everything you compile ends up in the self-contained environment. Furthermore, distributing your build products to another user (or another machine) is as simple as:

tar -czf mystuff.tar.gz ${PREFIX}

Then your friends can take your tarball and unzip it in any directory of their choosing. The whole prefix (environment) is self contained (except glibc, which must be at least as new as the version on your build machine). It just works.

View on HN

A small correction: if you use our linux compilers, then the software you build will link to glibc 2.12 and will run on any Linux with glibc 2.12 and above.

BTW we also have our own macOS clang 4.0.1 based compilers and they both run on and generate software for macOS 10.9 and above. You need to provide your own macOS 10.9 SDK and point CONDA_BUILD_SYSROOT to where you unpack it (or use one you got with Xcode and risk lower backwards compatibility).

> your friends can take your tarball and unzip it in any directory of their choosing. The whole prefix (environment) is self contained

How does this work if you are using ${PREFIX} with -rpath? I thought one had to use '$ORIGIN' to get a dynamic path that is relative to the executable's location. Unless conda is doing some '$ORIGIN' magic under the covers?

If you use conda-build it will do rpath fixups for you. You can also elect to specify that certain dependencies should come from the system. We do this for X11 for the Anaconda Distribution. To use these, in the recipe you list `cdt` (stands for Core Dependency Tree) packages as build dependencies. These packages are repackaged CentOS6 library and devel packages that are never installed on end-users machines, only at build time.

> If you use conda-build it will do rpath fixups for you.

Interesting. I'm not a big fan of magic like that, but I'll check it out anyway.

The magic has a lot of similarities to how nix packaging works. Using patchelf for example. Both nix and the conda communities have been at this for years, so there are large user bases with several years of production battle testing behind the techniques we use. For reference, we estimate a current 6M+ Anaconda and conda client users.

Why not build with -rpath $ORIGIN/../lib etc? Wouldn’t that make the rpath fixups unnecessary? The whole tree should remain portable.

Well, the Anaconda Distribution and conda-forge build many thousands of packages and they use a variety of (often programmable) build systems.

So quite often the build system will be mis-programmed or will simply not care for relocatability at all, hard-coding /usr/lib/libfoo.so as DT_NEEDED, or hard-coding /usr/lib as DT_RPATH and/or DT_RUNPATH.

So conda-build runs a post-build step to make all DSO loading relative. We use patchelf and install_name_tool for that on Linux and macOS respectively.

Do you have any links to docs or blog posts about how you'd go about doing that with Conda and ${PREFIX} managing compilers and libs? I'd love to see what a fully worked example looks like.

Official blog posts:

https://www.anaconda.com/blog/developer-blog/improved-securi...

https://www.anaconda.com/blog/developer-blog/announcing-the-...

https://www.anaconda.com/blog/developer-blog/utilizing-the-n...

I would recommend using conda-build to build packages, since it provides a lot of tooling to ensure things work well and to detect problems.

You can still create software by taring up the final ${PREFIX} if you wish, but if it's open source stuff then the best thing to do is to try to get the conda build recipes accepted by conda-forge.

The recipes that make the Anaconda Distribution can be found at: https://github.com/AnacondaRecipes/aggregate

Reply on news.ycombinator.com