Newer C++ features can create a lot of system yak shaving
rachelbythebay.com
rachelbythebay.com
I'm specifically stating this (obvious observation), because the headline might lead someone to believe that the features are at fault. The features are great and should considerably reduce yak shaving.
Going a step further, we've all been there. I once spent weeks building STLPort and boost, so I could avoid re-writing and extending an existing project against a C++ compiler from 2006, and more importantly, the WinCE 6.0 API (with weird timers and everything - ugh), and use a kind-of portable, recent version of C++, with extended features. I deeply questioned every single professional decision I had made the prior years. This had nothing to do with yak-shaving, but with how much platform-specific code you would end up prior to C++11, and how much simple, every-day functionality you had to write again and again. C++11 and later are a huge leap in this regard IMO, and for everyone stuck with an old compiler: Consider using boost.
However, ABI incompatibility is a real pain in C++ and one of the main reasons why I don't judge anyone building C libraries instead. While RHEL6 is just one example, you'll run into this issue all the time when developing on Windows. Here, either your C++ compiled libraries match 100% with the compiler used for to compile your application or it will simply not work. In 99.99% of the time you get a symbol issue with std::string you can be certain it's runtime library issue/compiler incompatibility.
Toss in there, I also have to compile a subset of libraries for Windows. Even when you use frameworks to isolate yourself from OS nuances, you may still have compatibility issues. Like, using the ACE library. What does the function you pass to ACE_Thread to spawn return? On posix systems, it returns void*. So, if you want to suppress compiler warnings (-Wzero-as-null-pointer-constant), you return nullptr. Oops, now your code breaks on Windows, because ACE_THR_FUNC_RETURN there is a DWORD, and no conversion from nullptr to DWORD. On yet other systems, it returns int. So, how do you have your warning free code on all the systems? Ugh: preprocessor.
But it is related to C++. When Go releases an update, I can grab the new compiler and just compile my stuff and the resulting binaries have very little dependency on the runtime (I may need to use the right libc or libssl version, but that's a job for Docker).
"But that's only because Go bakes the runtime into the binary!" - Exactly. - "Okay, that may make stuff easier, but it has huge disadvantages!" - I'm aware. The point is that the "system yak shaving" (as the submission calls it) is very much related to C++ and its features and how C++ compilers implement these.
"not related" might have been some bad wording, but the point is that it's not an issue of the C++ language or any C++ features, but it's because of how the tooling and libraries for C++ have been invented.
(And because there's no standard ABI.)
Make sure you use the same standard library for all components because libstdc++ and libc++ are different implementations with completely different object layouts. Usually libstdc++ is the system standard library implementation on most Linux flavours (i.e. it is part of the OS ABI), so unless you have specific requirements, go with that (specifically, it should work fine with clang).
libstdc++ ABI changed between C++03 and C++11 (that's probably the issue you are having with the string), but even in C++11 mode it is possible to use the old ABI (it is a compile time parameter), if you are willing to put up with the old reference counted strings, O(N) std::list::size and other minor things. If you can it is better to move the the new libstdc++ ABI of course, because the old one might not be maintained forever (it has been unchanged since GCC 3.6, though).
Windows is a pain because while the compiler ABI itself hasn't changed in a very long time (although undocumented I believe it has been fully reversed engineered), VC++ doesn't promise a stable library ABI and in fact it has breaking changes every major release.
https://docs.microsoft.com/en-us/cpp/porting/binary-compat-2...
Then you can usually get around all of the incompatibilities.
Even old proprietary UNIX compilers had their own set of compatibility issues.
Atlas, would it be that simple. A few years ago I was in a similar situation to the one described here, and I tried to switch to Boost. Things went bad piety soon, as this old post of mine described:
My internal monologues when writing modern C++ look a lot like this...
So I have to return from this function... it uses a lot of resources, so I don't want to copy it. But then there is RVO... Although is this actually NRVO? What was that rule again? I guess I can rely on move semantics? But only Scott Meyers himself understands those and he promptly exited the C++ community after figuring them out... And anyway, what if I want to change the ownership explicitly? Let's just use a unique_ptr. Oh but actually I mostly want to share it when I have returned, and that means two blocks of memory will be allocated per pointer instead of the optimal "single" allocation with the reference count at the beginning. Although maybe that's only in libc++? oh, and what if there is an error, should it return nullptr in that case?..
My joke was "Oh, I should return a unique_ptr<>, but then I will be losing out on the make_shared optimization" ;)
Like your hypothetical train of thought, it just goes to show that you can complicate C++ usage a lot more than necessary. Despite all the fancy new stuff for dealing with pointers and move semantics (and lvalues, rvalues, so-called universal references, etc.), the old standby of raw pointers still do the job just as well as they always have. You can always make some kind of note to come back to something later and decide whether it makes sense to use some other kind of pointer.
Huh, TIL http://scottmeyers.blogspot.fr/2015/12/good-to-go.html
EDIT: He was invited as a keynote speaker for DConf 2017: https://youtu.be/RT46MpK39rQ?list=PL3jwVPmk_PRxo23yyoc0Ip_cP...
were you running on potato computers ? I clearly remember building GCC 5.0 from source a few years ago and don't remember it taking more than 15-20 minutes on a i7 2600. (Of course you have to disable the whole unnecessary bootstrap stuff and only build c & c++ support).
$ repoquery python27 rh-python36-python
python27-0:1.1-25.el6.x86_64
rh-python36-python-0:3.6.3-1.el6.x86_64edit: well, OK, the official answer is more nuanced: https://access.redhat.com/documentation/en-us/red_hat_develo...
It's also strange to be unwilling to fork in source control, if necessary, a maintenance version for legacy servers and a current version for current C++ and current servers.
New software should probably run on more current servers and access the old RHEL 6 servers through stable network services (shared folders, DBMS, etc.) instead of expecting them to run bleeding edge software.
The mess I'm used to seeing on HPC systems with combinatorial builds, with everything done through a confusion of environment modules, bothers me and typically confuses users, especially when there's a system package that provides the same thing. (At least look at TACC's Lmod instead of the canonical modules implementation.)
I only had to deal with RHEL6.x that had gcc 4.8 installed via packaging, I think Red Hat had a solution for this.
Yes, Red Hat has a solution in the form of the Developer Tool Set, which will give you gcc 7.2.1[0]
[0] https://access.redhat.com/documentation/en-us/red_hat_develo...
Yes, it means you're using your own "mini-me environment", but you can share that environment across all of your C++ projects. As long as you build with:
LDFLAGS="-Wl,-rpath,${PREFIX}/lib -L${PREFIX}/lib"
... then everything you compile ends up in the self-contained environment. Furthermore, distributing your build products to another user (or another machine) is as simple as:
tar -czf mystuff.tar.gz ${PREFIX}
Then your friends can take your tarball and unzip it in any directory of their choosing. The whole prefix (environment) is self contained (except glibc, which must be at least as new as the version on your build machine). It just works.
BTW we also have our own macOS clang 4.0.1 based compilers and they both run on and generate software for macOS 10.9 and above. You need to provide your own macOS 10.9 SDK and point CONDA_BUILD_SYSROOT to where you unpack it (or use one you got with Xcode and risk lower backwards compatibility).
How does this work if you are using ${PREFIX} with -rpath? I thought one had to use '$ORIGIN' to get a dynamic path that is relative to the executable's location. Unless conda is doing some '$ORIGIN' magic under the covers?
So quite often the build system will be mis-programmed or will simply not care for relocatability at all, hard-coding /usr/lib/libfoo.so as DT_NEEDED, or hard-coding /usr/lib as DT_RPATH and/or DT_RUNPATH.
So conda-build runs a post-build step to make all DSO loading relative. We use patchelf and install_name_tool for that on Linux and macOS respectively.
Interesting. I'm not a big fan of magic like that, but I'll check it out anyway.
https://www.anaconda.com/blog/developer-blog/improved-securi...
https://www.anaconda.com/blog/developer-blog/announcing-the-...
https://www.anaconda.com/blog/developer-blog/utilizing-the-n...
I would recommend using conda-build to build packages, since it provides a lot of tooling to ensure things work well and to detect problems.
You can still create software by taring up the final ${PREFIX} if you wish, but if it's open source stuff then the best thing to do is to try to get the conda build recipes accepted by conda-forge.
The recipes that make the Anaconda Distribution can be found at: https://github.com/AnacondaRecipes/aggregate
This industry is just one big test of how long you can shave yaks (aka pay technical debt) before you quit and move to the forest. It's rewarding... until it's not. :)
Binaries created on RHEL 7 does not run on RHEL 6 - even when linking statically appears to work, you run the risk of running into the unforseen.
You could develop on RHEL 7 and build for production on RHEL 6, but this causes you enough grief and trobleshooting nesting up in all kinds of minor differences to not make it worth it.
Best you can do if you cannot upgrade from RHEL 6 is to install devtoolset(https://www.softwarecollections.org/en/scls/rhscl/devtoolset... - these are official packages crerated by Red Hat) to get a newer compiler.
I've been there, and it's not fun. And, been down the path of options the author enumerates after this statement.
Short of it is, on Linux (and I'm sure it's likely the same on BSDs & Mac), if you're not using the system supplied compiler, you're in for a world of hurt. You will have to build everything you depend on from scratch with your custom compiler. That said, it's possible. Just not fun, or user-friendly.
But all pointers do this in C++? Why would you need a smart pointer to do this for you?
I've written about this (http://btorpey.github.io/blog/2015/01/02/building-clang/), specifically in the context of getting clang running on RH6, which required first getting a C++11-compatible gcc running on RH6. For those who are in the same boat, you may find the article helpful.
10 times? Do you mean 10 platforms vs 1 on something like Java?
It's not enough to mandate how objects are laid out. People want the ability to define a library function that uses standard types - for example std::string - in the interface. To build a library with libstdc++ and link it with an application that has been built with libc++, there would (among other things) have to be a common string type that they agree on.
Herb Sutter has done some work in this area, but I don't know the current status of it. https://isocpp.org/files/papers/n4028.pdf
well, then there would be a single standard library implementation. I think that would be the way to go : just put libc++ out of the std namespace and in a custom namespace, `st2` for instance, and use its types from everywhere. MSVC, GCC, whatever.
https://access.redhat.com/documentation/en-us/red_hat_develo...
Or switch to Windows: Here you can use about any version of C++ you want (just e.g. install the respective version of Visual Studio that supports your desired version of C++). No need to pay attention which version of C++ is supported on which Windows version. :-)
And on Linux you can install newer compilers too.
The problems are still there; the root cause is C++ having no ABI, not Linux vs Windows.
But the original story, as far as I know, comes from Seth Godin http://sethgodin.typepad.com/seths_blog/2005/03/dont_shave_t...
Source: https://en.wiktionary.org/wiki/yak_shaving
Like when you have to spend hours fiddling with $package_manager because you want to build a $hipster_language app.
" 1) Any apparently useless activity which, by allowing you to overcome intermediate difficulties, allows you to solve a larger problem.
2) A less useful activity done consciously or subconsciously to procrastinate about a larger but more useful task. "
The virtualenv concept makes more sense: set a default on day 1 but assume you’ll want a new foundation periodically. When the foundations themselves are versioned, you can evolve your environment to some degree without having a completely chaotic mix of the latest updates.
Hence the Amazon build tooling for C and C++. I wish they would open source it, write about it, make it into an AWS product, or release it in any other form.
I'm sorry you guys still have to go through all this pain, I know it's a really hard problem with all the legacy.
And a bit more about Apollo as part of the AWS CodeDeploy announcement: https://www.allthingsdistributed.com/2014/11/apollo-amazon-d...
Isn't this just another day in the office for a C++ dev?
Just use docker?? That is literally what docker is for. Get your favourite flavour of debian or whatever, with a modern GCC and fire up that docker container.
EDIT: Oh wow, docker is only available in RHEL7 and higher. Damn.
Not that I necessarily recommend doing this.
Or even worse, they link happily then fail weirdly at run time because all the STL structure layouts have changed.
This is hugely exacerbated by Linux distributions perpetuating the fiction that C++ is an appropriate language for system library APIs. So now there is a generation of dependency-oriented developers who haven’t realized that “apt-get install libfoo-cxx-dev” is an enormous headache waiting to happen.