Interactive workflows for C++ with Jupyter
blog.jupyter.org
blog.jupyter.org
C++ on Jupyter looks amazing, and I really hope it makes starting with C++ easier.
However, if you're using conda, you might be able to at least simplify your build scripts, even though you can't eliminate them.
If you are okay with requiring your users to have conda, then you can exploit that fact to simplify your CMakeLists.txt in some ways. For instance, find_library, etc.. can be replaced with hard-coded links to ${CONDA_PREFIX}/lib/...
(I'm not saying that's necessarily "best practice" for all projects, but it's a nice option to consider, especially in the early phases of development.)
$ conda install conda-build # prerequisite...
$ https://github.com/conda-forge/fftw-feedstock && cd fftw-feedstock
$ emacs recipe/build.sh # Add --enable-openmp
$ conda build recipe
$ anaconda upload -u mychannel ${CONDA_PREFIX}/conda-bld/*-64/fftw-*.tar.bz2
In all of these I like the single "BUILD" file per directory, and simple text file explaining dependencies.
Also these tools work awesome when you are in a company, and you have underneath - dozen or more libs, sdks, projects that need to run together.
I'm pretty sure it works great then. But for the bigger C/C++ ecosystem the problem is that everyone uses another build system. From Makefiles to Automake, Cmake, Gyp, Scons, Bazel, etc. everything is included. As soon as you try to use dependencies that favor a different format you either have to rewrite their project definition in your format or at least build it as an external subproject and manually add the include directories.
Imho that's one of the biggest weaknesses of C++ that even outweighs language concern for me: It's often lots of work to integrate 3rd party libraries. And because of this there isn't a really great ecosystem of libraries (e.g. compared to Javascript and Java).
The C mentality of some devs dragged into C++ world.
Already in the 90's we had very nice high level C++ libraries, that could compete with what Java offered later.
Turbo Vision on MS-DOS, Object Windows Library and Visual Components Library on Windows. All from Borland.
PowerPlant on Mac OS, from Metrowerks.
From Microsoft, we had MFC, which started high level like those ones (originally named Afx), but then the beta testers requested for it to just be a thin layer over Win16 and it was reborn as MFC.
Many of the modern C++ patterns were already possible with those libraries, but the C wisdom made many not use them.
Another modern example is Android NDK, Google writes the code in nice C++ classes, that get exposed as low level C APIs, or have a Java JNI barrier (e.g. Skia).
I'm not on the android team, but in general it's easier to maintain reverse-compatibility with C ABIs than with C++ ABIs.
And the plethora of libraries and the modest size of the standard library are part of the problem. Let's say you're writing a C++ library that understands git and lets users develop more complex applications with your libgitxx as a building block. Whose filesystem primitives (file, directory, path, user, etc.) do you use? There's really not a satisfactory answer to that question.
Point being, C ABIs don't have this problem as much. They tend to just pass around ints and char. Maybe there are some structs to pass around, but they tend to be made up of ints, char, etc.
As such, backward compatibility (binary-wise) is not needed.
Android, Chrome, etc. are another matter - former has an SDK, and I guess you may need that there, but I don't have much experience with it.
In Android's case, the ABI would be whatever g++, now clang, shipped with NDK support.
Additionally there are ways to implement compatible ABIs in C++.
In any case, they are wrapping the C++ classes already, either via C or Java, so they could offer them as well.
Instead, one needs to write unsafe C code, JNI boilerplate or wrap them again in C++.
> Additionally there are ways to implement compatible ABIs in C++.
Eh... when even compilation flags affect ABIs, it's hard for a particlar C++ file to have a stable ABI. The same is true in C, of course, but the way C libraries are generally designed, it's simpler to be ABI compatible. In C++, arcane and unexpected things like noexcept specifications (affected by changes in underlying libraries!) can change your ABI.
> Instead, one needs to write unsafe C code, JNI boilerplate or wrap them again in C++.
That's a fair point. There might be reasons they'd do that (SLAs they can afford), but that's a presumption.
Which circles back to my statement about "The C mentality of some devs dragged into C++ world.".
Many useful C++ libraries were not standardized in the early days beyond STL, because many thought C++ should be like C where libraries are whatever the OS provides and nothing else.
Which is clearly visible in the decision to have ANSI C and POSIX APIs as two separate standards, although most C compilers try to provide some kind of additional POSIX compatibility, specially visible in non-POSIX OSes.
Thankfully, the ANSI C++ committee realized the mistake and now we are getting a useful set of libraries into the standard.
As for compilation flags, even for C it is only true if all compilers on the platform follow the same ABI.
And you still need to provide all variants anyway (debug, single and multithreaded, hard and soft float, pre-defined pre-processor macros, ...).
C ABI only appears to be simple in OSes that happen to be written in C.
Grated that is the case nowadays with most being UNIX clones, besides Windows and mainframes, but it wasn't always like that.
I remember the pain of trying to mix multiple C compilers back in the 90's.
> C ABI only appears to be simple in OSes that happen to be written in C.
I'd say that C is the only language that (mostly accidentally and non-standard, for the reasons you state) provides a simple a (relatively) stable ABI.
Apparently something related to C++ APIs on the NDK is planned to start appearing with NDR r17 onwards, as per Github issues.
Today, Qt, for example has qmake, cmake, qbs? others?
The bazelment trunk, provides bazel rules for building ~30 libraries, among them: openssl, zlib, v8, folly, boost, and others: https://github.com/bazelment/trunk/tree/master/third_party
At work, I'm heavily vested in using Microsoft's VCPKG for new builds of open source libs, they internally use CMake, but then CMakeLists.txt were not even readily available for some of these projects, and the VCPKG maintainers did them.
At some point, or it may already exist, someone would make a converter from one system to another, lossy possibly. For example it's possible to parse everything you need, correctly from a .sln/.vcxproj usign the MSBuild framework, which is now available everywhere (except the actual .props, .target files, which are Visual Studio specific). But on a machine with the Community version of the compiler, these are free to use (should even work under wine). And since a lot of other build systems target .sln, .vcxproj files, this could in a way be a target for a lot of systems to converge, and from there extract what you want: - List of source, header files. - Separation between projects - Defines for each project. - etc.
It doesn't have to be perfect, but it's possible.
Or maybe from ninja generated files (though not meant to be used that way).
Luckily there is sane way to verify whether alternative build for a project works - simply compile it, and verify whether the produced artifacts are the same, or are close enough (list of produced header files, exported symbols, may go even deeper).
And yes, I do miss the Turbo/Borland Pascal's .TPU files, where no .h file was needed, or Borland C++ simple project file that was just listing line by line the .c files, but back then projects (which I was familiar with) were much smaller, even gcc, the linux kernel and others were much smaller then.
It only seems difficult if you're stuck in the mentality of shipping all your dependencies with your source code and statically linking everything. C and C++ libraries actually care about maintaining API (and usually ABI) compatibility, so you don't have to ensure you have a specific version of dependencies.
In C, you still had incompatibilities between compilers. (MS fastcall convention and declspec come to mind...)
Like any teenager tinkering, I opened up some sample cpp file that came with the compiler and started just throwing in cout<< all over the place doing random stuff with variables. My older brother explained "loops", "functions" and later the basics of "arrays" and I completed my first game by 16, written purely in C++ -- a Tank clone that was a single long-ass cpp file using purely #include<graphics.h> calls, putting individual pixels on a 640x480 screen. I hadn't learned what classes were yet, so it was some form of "functional programming", just starting from a void main()
The point was, it was easy to tinker and learn and make stuff and explore in C++, the language isn't innately arcane. I didn't even have the internet back then to look stuff up. A teacher gave me a textbook for "graphics in C++" because my high school (which I wasn't in yet) computer science curriculum didn't cover graphics, but I hated copying code from the textbook (it seemed outdated) so I just opened up header files and tried to call functions I saw in there. I didn't need to use any fancy C++ syntax, just the basics my brother taught me. No make files, no build scripts, no gcc. Just an IDE that existed on a floppy disk and the most common OS available at the time (DOS).
It must be a sign I'm getting crotchety and old, but "back in my day" C++ wasn't such a confounding beast. And I'm sad to say I think modern-day python is going down a similar path, it started out intending to be the easiest language to tinker in, but it's slowly becoming arcane just to import a module and launch a program for a beginner.
I stepped back into C++ recently trying to make something "I can see" at from scratch, and it's damn nigh impossible without visiting 4-8 different instructional sites depending on how you define "something I can look at".
Yet the naysayers state that apparently we can have standard IO, filesystem and networking support, but not 2D.
The reason being that not all scenarios have graphical displays, as if all deployment scenarios always have filesystem and networking stacks.
'Procedural' is the term you were looking for.
I wanted to write a Postgres extension/fork, and was thinking "oh I'll just replace some bits with Rust".
Unfortunately everything is `make`, which is great when you're setting something up and have a huge process... but it's very hard to make some changes without figuring out the entire flow.
Someone that could somehow wrap make to make it easier to make incremental changes on existing projects could help fuel a lot more experimental changes.
I think there is a deep truth in (unintended?) pun: those who don't understand make are bound to reinvent it, poorly.
It's not the worst thing to ever grace the planet by a long shot, but I think it suffers from the same problem that bash scripts have. Everything is string-ly typed and you can only build up a thing to a certain level of complexity easily.
There's of course a lot of requirements in systems software, of course. And a lot of difficulty stems more from the C ecosystem's difficulties with packaging (where are the C equivalents to "environments" you find in Python/Ruby so you don't have to pass every lib in explicitly?)
I think there's some useful opportunities somewhere here.
[0]: https://github.com/postgres/postgres/blob/master/src/Makefil...
Yes, make was designed for the 1970 world, where all you care about is building simple utilities on Unix. There was no today's complexity (out of tree builds, cross-compilation, etc) nor "alien" platforms like Windows where people have spaces in their paths.
What we need is to redesign make to handle today's requierements. And this is what we are trying to do with build2.
Written by Matt Hermann who I believe is now at google - I've never met him. I worked somewhere that used it and have used it ever since for every greenfields C++ project I've done and just don't think about my build system anymore. For me this just works for everything I need, adding a linkflag for this .cpp file only? Dead simple, hard to imagine it being simpler. Build tests and run with valgrind equally so. I have a single top level makefile that literally just specifies targets, eg release, test, asan_test, clang_release etc. Each is a single cake line.
I highly recommend it for C++ on linux (haven't used it elsewhere to say), for me it's a thing of beauty and a joy forever. Not dealing with autotools or scons or complicated make based build systems makes C++ so much nicer to work with.
But you do this precisely once.
- I have to include third party libraries occasionally, and sometimes weight if it wouldn't be just easier, faster and better for my mental health to reinvent the wheel instead.
- I have to check out, test or use third party projects every now and then. They all use different build systems, different ways to use different build systems and different work around to get around limitations of build systems.
I like C++11 and love C++17 but absolutely hate building and managing C++ projects. It's a mess.
Ugh
One thing I disagree with is the idea that the lack of a good interactive environment for C++ makes it difficult to teach. C++ is a compiled language, so learning how to compile a C++ library or program is _part_ of C++. I feel like C++ beginners should go interactive _after_ learning how to compile a project. I write that with first hand experience; my first programming experience was with writing interpreted C++ in CINT and I feel like it hindered my ability to eventually understand what a real C++ program was.
So having a good interactive environment is really useful for the first lessons, then you can teach the compilation chaintool and switch to a real environment.
Interactive development has immense benefits in exploring apis for experts and learning the language for beginners. And C++ is not well-known for its compile-times.
With regard to the full stack, I tend to agree with you, at least in the academic environment. Organizations that don't have a lot of software engineers will shoot themselves in the foot if they focus on the C++ language rather than the infrastructure around it. An interactive notebook is great to teach someone to add a few lines of code here or there, or to run quick tests, but what we're lacking is people who understand how to plug their jupyter notebook into the production line.
The issue is developers having open mind to such environments, luckily it seems to be changing with QtCreator, CLion, VC++, specially thanks to clang being implemented as a library.
I entirely disagree. This is only for historical reasons, the language itself does not care at all about the compilation model. This mindset is what is keeping C++ back, both for the industry AND in students' minds.
> I feel like it hindered my ability to eventually understand what a real C++ program was.
The moment you have a .o file it's not a C++ program anymore, but a platform-specific object file. You aren't learning C++ but windows / mac / linux's native binary production toolchain.
Then that is at least part of the learning curve of seriously using C++ so I think as of today it's (potentially) dangerous to start with only interpreted code and wait to be introduced to compiling, build tools, etc. They should be taught together. I also think it's not the greatest environment - and I hope it changes, perhaps projects like this will help!
You get to build the object file, shared objects, dlls what have you. Then use the linker to link it to turn it into an executable.
Some modern languages specify the runtime very deep. For example Python or Java - so much that it is hard to separate language from runtime or standard library.
That being said, I look down on all the commentators who say that C++ has a primitive build system/package manager etc. Lets be clear: C++ has no package manager, and , in my view of the world, no compiled language has a native build system, its just that most compiled languages dont separate compilation from linking and also most languages support modules :)
C++ is hard. Throwing some fancy frontend and and a REPL will only make you realise that sooner. The complexity is mostly unjustified, but the language is so powerful that you could do anything with it: from trading strategies to power plant control systems to game engines. I have been studying C++ for 9 years, have used it for 7, and have been a professional developer for only 2.5 years, and I think I have only scratched the surface. I have used python for a lot less time but I feel like I am more comfortable with it because Python is trivial to learn (to a person with a comp sci background ofc). What you think you are cutting away (dependencies/build jnfo) is something essential to learning C++ and how everything comes together. Better learn it sooner than later imho.
Rant over
Most modern, mainstream compiled languages have a build system that wraps around the compiler and linker. Invoking the compiler and linker are responsibilities of the build system, and frequently modules are compilation units.
Modern C++ is a neat language (I used to be a C++ developer) but it's tooling alone makes it unsuitable for any project where rapid iteration is remotely important.
Modern C++ when combined with Qt makes rapid iteration very doable. In my opinion Qt should be the standard c++ model.
The consistent style is actually something nice. However lots of C++ programmers will not necessary like the OO style because it will lead to relatively slow programs. I guess with all the indirection which is going on and the metadata-based reflection and signal/slot system it might not really be ahead of even easier to use managed languages, which are even more convenient to use.
If you look at the Boost (or also C++11/14/17) libraries you will often find a very different programming style, which is heavily based on templates. This one is a lot harder to read and understand for non-experts, but it allows to achieve an even higher performance. The authors might think that this is the preferred model for C++.
I was fighting against QuantStack's modified software stack for two hours until I discovered this:
https://github.com/QuantStack/xeus-cling/issues/74#issuecomm...
Fortunately, they updated the README.md in
I would like to print it in very big letters and put it at the entrance of slow-moving IT departments. "cost of change" is likely one of the most important metrics to measure the effectiveness of a team.
Being risk adverse because of fear of change leads to stagnation.
"more Jupyter notebooks and less spreadsheets" could be a useful simplification for conveying this change of POV.
Worse, so called "notebooks" are neat, but they are just as bad for engineering best practices as spreadsheets are.
Not to say that you shouldn't be able to grow from freeform practices. You should. Just don't mistake them for being somehow superior to spreadsheets.
Luckily, the performance constraints of most environments are such that python is not an automatic deal breaker nowadays. That said, correctness of code proofs are usually different from correctness of machine learning algorithms. Such that mixing them seems to just fool both sets of practitioners.
Correctness of machine learning is a complicated problem generally.
https://github.com/hal88/junkcode/blob/master/template_scrip...
I post in the hope that it is a useful idea to someone and also that people respond with their similar ideas that might be useful to me.
I only wish it had come a few months earlier. I've had to switch from python in a jupyter notebook to C++ for performance reasons, and I really felt the loss of incrementalism.
However, I'll essentially never be able to run my code from a notebook because it runs on a cluster I need to SSH into. Thing is, I'd like to develop that code locally and iteratively.
Thus it was required to do a lot of incremental development in C++. It wouldn't make sense to add in a manual transpilation at every increment, and I needed the actual C++ code to judge performance.
Install a c++ compiler, create main.cpp, `g++ main.cpp`
What am I missing?
The students don't know anything about the CLI. They might not know what a directory is or how to cd to it. They may have to install multiple packages. On certain operating systems they'll have to set some PATH variables.
I'm not saying it shouldn't be that easy, but due to failures in earlier education it's not.
Once we had the basics down they introduced Linux, the command line interface to GCC, etc.
Tried also to clone the github repo and compile locally but got tired of fixing dependencies and stuff after a few compiler errors.
QuantStack/xwidgets https://github.com/QuantStack/xwidgets
QuantStack/xplot (bqplot) https://github.com/QuantStack/xplot
Some problems aren't suited well for it though, especially those that involve Python dictionaries or sets. For those problems, switching to C++ results in MUCH faster code (std::unordered_map is way faster than Python's dict, especially for small elements).