amadio

Most software at CERN is open source. But you are right, for things which are quite specific to CERN, like the control systems of the accelerator, there's also some proprietary code. Same for hardware actually, CERN develops open hardware too.

https://opensource.web.cern.ch/

I'm a Gentoo dev and work at CERN since a while. Of course I jokingly suggested that to my colleagues. I think it could actually work pretty well, though. Many physicists use Gentoo.

We run most things on Alma Linux, but there's also RedHat in the mix.

A good intersection of CERN and Gentoo is EESSI. They use CVMFS, which is a FUSE filesystem used for software distribution from CERN, and Gentoo as part of their software stack. See links below for more info:

- https://www.eessi.io/docs/filesystem_layer/

- https://www.eessi.io/docs/compatibility_layer/

Related article: https://doi.org/10.1088/1742-6596/513/2/022032

These da Vinci style drawings were likely created using the SketchUp models from sometime ago, which you can download from the page below to play with:

https://twiki.cern.ch/twiki/bin/view/CMSPublic/SketchUpCMS

If you would like to know more details about the physics case for FCC, there was an academic training recently at CERN:

https://indico.cern.ch/event/1582427/

For those of you using CMake, have a look at the module below:

https://github.com/xrootd/xrootd/blob/master/cmake/XRootDVer...

and also the genversion.sh script at the top of the repo.

I use these plus #cmakedefine and git tags to manage the project version without having to do it via commits.

At CERN, software stacks are created centrally and software distribution for experiments is done with CVMFS (https://cernvm.cern.ch/fs/), an HTTP-based read-only FUSE filesystem.

EESSI (https://eessi.io) has taken this model further by using CVMFS, Gentoo Prefix (https://prefix.gentoo.org), and EasyBuild to create full HPC environments for various architectures.

CVMFS also has a docker driver to allow only used parts of a container image to be fetched on demand, which is very good for cases in which only a small part of a fat image is used in a job. EESSI has some documentation about it here: https://www.eessi.io/docs/tutorial/containers/

For those interested, there's more information here: https://home.cern/science/computing/birth-web

If you come to CERN, you can also see the NeXT machine that was the first web server.

CERN is a heavy user of ceph, with about 100PB of data across cephfs, object stores (used as backend for S3), and block storage (mostly for storage for VMs). CVMFS (https://cernvm.cern.ch/fs/) is used to distribute the software stacks used by LHC experiments across the WLCG (Worldwide LHC Computing Grid), and is back by S3 with ceph for its storage needs. Physics data, however, is stored on EOS (https://eos.web.cern.ch) and CERN just recently crossed the 1EB mark of raw disk storage managed by EOS. EOS is also used as the storage solution for CERNBox (https://cernbox.web.cern.ch/), which holds user data. Data analyses use ROOT and read the data remotely from EOS using XRootD (https://github.com/xrootd/xrootd), as EOS is itself based on XRootD. XRootD is very efficient to read data across the network compared to other solutions. It is also used by other experiments beyond high energy physics, for example by LSST in its clustered database called Qserv (https://qserv.lsst.io).

I think this is good advice overall. I wrote a CMake script that does most of the heavy lifting for XRootD (see https://news.ycombinator.com/item?id=39657703). The CI is then a couple of lines, one to install the dependencies using the packaging tools, and another one calling that script. So don't underestimate the convenience that packaging can give you when installing dependencies.

Since you left this "optimize me" comment here:

https://github.com/RandyGaul/cute_headers/blob/755849fc2819d...

See an optimized quaternion multiplication implementation in SSE by me here:

https://stackoverflow.com/questions/18542894/how-to-multiply...

I was just taking a look and couldn't help but notice the switch statement for your operator[], which likely causes a lot of unnecessary bad speculation at runtime:

https://github.com/RandyGaul/cute_headers/blob/755849fc2819d...

I fixed this exact problem in a highly used library in high energy physics:

https://gitlab.cern.ch/CLHEP/CLHEP/-/commit/5f20daf0cae91179...

Many believe the C++ compiler will magically optimize the switch away, but in some cases, like the example above for CLHEP, it doesn't happen, so you end up with bad performance.

Unfortunately this is no longer maintained, but it's interesting nonetheless: https://abi-laboratory.pro/?view=timeline&l=curl

There is also a new format being developed for Run 4, RNTuple:

- https://indico.fnal.gov/event/23628/contributions/240607/

- https://indico.cern.ch/event/1338689/contributions/6077632/

Robin Green has some excellent material on the subject, which I am linking below.

Faster Math Functions:

https://basesandframes.files.wordpress.com/2016/05/fast-math...

https://basesandframes.files.wordpress.com/2016/05/fast-math...

Even faster math functions GDC 2020:

https://gdcvault.com/play/1026734/Math-in-Game-Development-S...

See also:

- Gentoo Prefix https://wiki.gentoo.org/wiki/Project:Prefix

- European Environment for Scientific Software Installations (EESSI) https://www.eessi-hpc.com/

- REANA (Reusable Analyses) https://reanahub.io/

It was a nice guest post on the website about eclipse, but most people just use gdb. It is now possible to step through ROOT macros with gdb by exporting CLING_DEBUG=1. See https://indico.jlab.org/event/459/contributions/11563/

ROOT 7 is coming. Things are being discussed this year about it, the target is for HL-LHC. See link below. https://indico.cern.ch/event/1369601/contributions/5867782/a...

I take issue with this part of the article:

> In general, managed tools will give you stronger governance and access controls compared to open source solutions. For businesses dealing with sensitive data that requires a robust security model, commercial solutions may be worth investing in, as they can provide an added layer of reassurance and a stronger audit trail.

There are definitely open source solutions capable of managing vast amounts of data securely. The storage group at CERN develops EOS (a distributed filesystem based on the XRootD framework), and CERNBox, which puts a nice web interface on top. See https://github.com/xrootd/xrootd and https://github.com/cern-eos/eos for more information. See also https://techweekstorage.web.cern.ch, a recent event we had along with CS3 at CERN.

Nice to see cliutils trending. Henry really understands CMake, so advice by him is always good to follow. I'm not a big fan of CMake, but after years of suffering, I mean, experience, I've learned how to use it relatively well. One feature I like is the "dashboard" model of ctest. It's somewhat painful to setup, but once you have it working, it makes your life building and testing your project much easier. Unfortunately, when I posted it nobody noticed, but if you want to have a really simple CI setup for your CMake project, checkout this link:

https://news.ycombinator.com/item?id=39657703

See the CI workflow using my script here: https://github.com/xrootd/xrootd/blob/master/.github/workflo...

Also, with some advice from Henry I wrote a nice new setup.py for the Python bindings of XRootD integrated with the CMake build. I didn't want a dependency in scikit-build in the end, but his advice helped me a lot.

For me, one of the biggest leaps in how I think about performance was when I learned about the Top-Down Micro-Architecture Analysis Method, by Ahmad Yasin from Intel. You can learn the main ideas from himself in the video below:

https://youtu.be/kjufVhyuV_A

The idea to classify cycles into front-end bound, backend bound, bad speculation or memory bound is brilliant. Once you know which one your program suffers from, it's easy to know what can be done to improve things.

I usually just set the working directory to ${CMAKE_CURRENT_SOURCE_DIR}¹ for tests that only read files, and use configure_file()² for files that could be modified or contain variables expanded by CMake. Note that you don't need to add a dependency, as configure_file() runs during configuration, and if you change the input, will copy the file again. CMake also has the notion of text fixtures for more complicated things as well, like setting up a db or other kind of service that your tests may need to run. See property FIXTURES_SETUP³ for that.

1. https://cmake.org/cmake/help/latest/prop_test/WORKING_DIRECT...

2. https://cmake.org/cmake/help/latest/command/configure_file.h...

3. https://cmake.org/cmake/help/latest/prop_test/FIXTURES_SETUP...

In VecCore (a small C++ SIMD abstraction library on top of Vc and std::simd), I created some simple examples to show how to use the library to optimize code using SIMD in a somewhat generic way. You can find it on GitHub at https://github.com/root-project/veccore

I have examples for Julia sets and the Mandelbrot set, including an implementation with AVX2 intrinsics.

These days with std::simd more widely available there's less of a reason to use VecCore, but the examples may still be educational enough. I chose Julia sets and the Mandelbrot since they are perfect examples of simple problems that compilers fail to vectorize on their own.

I think the idea of Framework is really good, but static data limits the applications, excluding monitoring and other cases in which the data is constantly changing, but the dashboard can stay as it is. For example, I'd love to see a revamped Framework version of the LHC beam monitor and related pages (see https://op-webtools.web.cern.ch/vistar/, but check again in 2 months or so, when the accelerator will be running).

In high-energy physics, ROOT is /the/ toolkit for data analysis, and I guess jsROOT (https://root.cern.ch/js/) could also be used to load data to be shown in Framework dashboards. I thought the idea of Framework as a blogging engine with powerful data visualization built-in could be very interesting. Think, for example, about physicists pulling open data (https://opendata.cern.ch) and writing about their analysis or someone pulling data from https://ourworldindata.org/ in their own visualizations to support their case while writing about a particular subject, etc.

Yes, let me expand a bit on the answer above.

EOS was designed and developed with the unique needs of the LHC experiments in mind. The advantages it has are features used by the experiments, like support for remote access via the XRootD protocol, which is used for data analysis with ROOT (only the parts of files needed by an analysis are downloaded); rich support for client authentication methods (Kerberos, X509, OIDC, etc); and support to also FUSE mount everything to give a convenient POSIX-like view of the data. EOS needs to sustain ingestion of data at high rates from experiments (10s of GB/s each) for several months at a time during data taking without any downtime, while at the same time having tens of thousands of clients connected reading data as well. It's also integrated with the CERN Tape Archive (CTA) and File Tranfer Service (FTS), used for long term archival and data management across sites, respectively.

In the cases where block/object storage is needed, like storage for VMs, S3 storage for various uses, etc, then ceph is better suited. It has lower latency and EOS does not offer block-level access. In addition to providing storage services for OpenStack/Openshift, ceph is used to provide storage to back AFS and CVMFS, for example. CVMFS is another interesting piece of CERN's infrastructure, it's a read-only, HTTP-based FUSE filesystem used to distribute the software used by the experiments to grid sites around the world. Dan van der Ster, mentioned in the article above, has a good overview of ceph usage at CERN here: https://youtu.be/2I_U2p-trwI?si=Tsq4h8NIu4vSZQwt

If you are interested in EOS, we have the EOS workshop coming up in March: https://indico.cern.ch/event/1353101/

EOS and ceph have different use cases at CERN. EOS holds physics data and user data in CERNBox, while ceph is used for a lot of the rest (e.g. storage for VMs, and other applications). So both will continue to be used as they are now. CERN has over 100PB on ceph.

EOS (https://cern.ch/eos, https://github.com/cern-eos/eos) is probably a bit more complicated than other solutions to setup and manage, but does allow to add/remove new disks and nodes serving data on the fly. This is essential to let us upgrade harware of the clusters serving experimental data with minimal to no downtime.

Nice article! We've also recently reached the mark of 1TB/s at CERN, but with EOS (https://cern.ch/eos), not ceph: https://www.home.cern/news/news/computing/exabyte-disk-stora...

Our EOS clusters have a lot more nodes, however, and use mostly HDDs. CERN also uses ceph extensively.

Just to complement what I said above, here's a link to a presentation I made a couple of years ago about performance optimizations in simulation: https://indico.cern.ch/event/1052654/contributions/4521602/

We also have meetings dedicated to performance, some of which are not public, but this series from ROOT is: https://indico.cern.ch/category/14122/ If you search above, you will see many discussions about performance. The CI for ROOT also has a set of benchmarks to catch regressions, and Geant4 has two systems to track performance, a CI job checking every merge request, which I've set up myself (not publicly accessible), and a more complex system to track performance run by FNAL: https://g4cpt.fnal.gov/

These are just some examples from the projects I've worked on. There are also efforts to port stuff to GPUs and HPCs, and many other projects like event generators that are also undergoing performance work for HL-LHC. If you Google you can probably find a lot more stuff than what I already mentioned. Cheers,

This is not true. Maybe a PhD student doesn't care much (or doesn't know), but we care deeply about software performance at CERN. I've worked myself on optimizations in detector simulation and data analysis software (Geant4 and ROOT) for a few years. Later in this decade, when HL-LHC comes online, the only way to be able to cope with the 10x increase in data rate from experiments and a matching increase in simulation requirements will be to optimize as much as we can the software we have, because we will not have the money to just buy 10x the hardware we have now.