Not a physicist, so maybe I'm misunderstanding...
How is smashing protons into each other at relativistic speeds billions of times not a 'brute force approach'?
How is smashing protons into each other at relativistic speeds billions of times not a 'brute force approach'?
Before they outlined what they were looking for. Now they're just sifting.
Most science doesn't allow you to propose "We will expose A to X and see what happens", but rather expects a hypothesis: "We expect Z following X>A because of [rationale]."
Which I consider one of the major flaws of the current scientific practice. There's absolutely nothing wrong with trawling over data looking for interesting things. You just have to be more statistically careful. Huge amounts of science have been done by just trawling over things or rapid-fire throwing theories at the wall and seeing what sticks. Having to always call your shots means you can only slightly push the frontier back. There's a place for that, which is probably "the vast majority of experiments". But we need that more exploratory stuff too.
But somehow we got entrenched with a terrible oversimplification of science as "The Way Science Is Done", which hyperspecifies the parameters of the ways we can modify our confidence values in various theories. There's more valid choices than is currently considered valid, and we're missing out.
Did you create a model and then smash them together and analyze them? So you are testing "I believe that if I smash protons at X energy level that they will cause y_0,...,y_n decays" This is standard hypothesis testing. You are looking in a specific place.
A brute force method in this area is more like "Well... I'm not really sure. Why don't we just smash these protons together at uhhh... idk X energy?" And then you get "well that was interesting" or "okay, that was expected".
This is overly simplified. Basically a brute force method is just doing and observing. Not really predicting things. You're searching a large solution space and so you just try random things and hope you get lucky. (they have some good guesses of areas too look, so it isn't completely in the dark)
Then "brute force" approach has to do with what can be found in that enormous data set and how physicists have, until now, been looking for specific patterns from theories and expectations.
Basically, the LHC spits out data so your your Big Data looks small. The first pass was for data confirming/disconfirming some big theories where a specific peek at the data would give meaningful answers. So far the data shows that we had it right to start with.
Now, for the next pass, they're expanding how they look at the data to find anything 'unusual'. "Pattern recognition" in this case is a bit harder than just checking for a recurring value, or a "true" in the "Unified theory of everything?" field. We're a bit closer to what we would generally call AI, and at very least some interesting techniques with more interesting followup experiments to weed out false positives.
The "general search" doesn't target a particular model. The only model used is our Standard Model, treated as background. Traditional searches utilize one or more of many Beyond Standard Model (BSM) models to treat as potential signal.
Communications and processing capabilities have increased substantially since then. One may now be able, with investment/expansion of those, to take a more "brute force" approach in mining closer to if not at the level of raw data and arbitrary -- or at least a broader range of -- associations.
All of this requires us to define what constituents an interesting event. The most basic trigger is called the "minimum bias trigger" because it requires us to make the fewest assumptions about what makes the event interesting. So even at the most general level a brute force approach will still have some bias because it will likely be using the minimum bias data. What makes it "brute force" is that they will no longer be looking for signals from specific new models, they will only be considering the Standard Model and essentially looking for outliers. The difficult part is that there is so much systematic uncertainty given the complexity of these detectors, which makes it hard to obtain statistical significance.
If it's really raw how much room is there for domain specific lossless compression?
How many seconds long has or will the experiment (CMS) normally run?
How much did or will the CMS component of LHC cost from construction to its full lifespan?
What fraction of that imaginary lump sum went to digital storage budget?
Compression is most definitely used for the data that is actually saved to disk but it is not done on the fly when trigger systems have to run in nanoseconds.
The data limitations are not really from a lack of foresight or proper budgeting. It was known decades ago how much data would be produced by the LHC, it just really is an insane amount of data.
[0] after applying triggers (i.e. filtering) only a few hundred petabytes of data is stored in a given year from the CMS detector.
I'm trying to understand your numbers, at 10TB/s and 40 million pictures per second, thats 250kB per picture. since there are 75 megapixel (or I assume 75 megasensors) how does that work for raw data?
Is the energy determined from calorimeters, or from curvature in magnetic field?
What exactly is the ADC measuring? deposited charge?
Since you are acquainted with the project, could you point me to a good pdf describing the detector and low level digitization architecture (pre FPGA) architecture?
What is the bit depth of the ADCs?
I'm sorry if I sound dismissive here, I don't mean to be it's just impossible to go into all the detail of these machines in an HN comment. They are unbelievably complex (and fascinating!). I encourage you to check out the CMS design doc that will cover most of it in extreme detail (it's almost 400 pages) [0]. There is also a much more general overview of the various detector subsystems that is considerably more concise [1].
To answer one of your questions though: the tracker is used to measure the track of charged particles through the magnetic field and determine their momentum. This isn't enough to tell us their energy though b/c we don't know how massive each particle is. This is why the tracker is surrounded by an electromagnetic calorimeter (which "catches" electrons and photons) that is surrounded by a hadronic calorimeter (which "catches" hadronic patricles like neutrons and protons). The information from these systems is combined to ID each particle and determine how much energy it had, whether or not it decayed, etc. This is generally how we study the physics involved in the collisions but in reality it is 100x more complicated than this and there are number of other detector subsystems involved.
[0] http://inspirehep.net/record/796887/files/fermilab-pub-08-71... [1] http://cms.web.cern.ch/news/detector-overview
I have the impression that the reason the data for a single event is so large is because it is ambiguous, i.e. the uncertainty of a possibly incorrect interpretation forces you to store a lot of analog values (so that the interesting events can be checked or re-interpreted later on), and this size is what forces you to store a lot of data, and this forces you to triage for interesting events.
Assuming this is correct, then if the uncertainty of the tracks and their energies could be decreased, the data would be much more compact (i.e. only store location, momentum, energy/mass of incoming and decay particles as opposed to all the possibly relevant for later analysis ADC values)
Assuming that this level of correctness could be achieved by solid-state track detectors [0], where the particles leave tracks throughout a solid material, would it not (in theory) make sense to 1) continuously pass a such a detector material above and below the collision point, 2) at high speed slice, 3) in massively parallel etch, 4) microscopically examine the accurate tracks (bubble chamber style) 5) digitally store them compactly by the above assumption that the event size in bytes would be much lower if we only needed to store few but exact parameters per event 6) remelt the preferably low melting point etched track detector material 7) feed the erased etched track detector back to the collision point in a continuous fashion.
Either the speed of the material should be quite high or the illumination zone or collision point engineered very compact. Different collision bunches would be disambiguated because the tracks point to a different origin point (since the material has moved in the 25 ns @ 40 MHz, so for sub micrometer collision point that is 40 meters per second! obviously the detector would benefit from being designed to steer the collision point so it alternates to different positions)
Obviously the biggest issue with this approach is needing a huge facility for parallel etching and inspecting the slices
the cooling time should not be an issue, since the circular buffer of remelted material could be made arbitrarily long
do you think it could make sense for a future detector to temporarily store the tracks physically in a solid-state nuclear track detector ?
[0] https://en.wikipedia.org/wiki/Solid-state_nuclear_track_dete...
Edit: I forgot to mention that energy could then be determined from the change in curvature as the particle loses energy
Here is a picture of a heavy-ion collision reconstructed from the CMS detectors to give you an idea of how much needs to be accounted for [0]. The light orange lines are the particle tracks in the tracker, the red boxes indicate the amount of energy deposited in the EM calorimeter, and the blue boxes indicate the amount of energy deposited in the hadronic calorimeter.
I should note that normal proton-proton collisions do not produce anywhere near this number of particles (what we call the event "multiplicty") but they can still produce a few hundred. Furthermore, there are, on average, 40 proton-proton collisions per bunch crossing (which is what dictates the 40 MHz rate). So every 25ns there is a bunch crossing and the detector "snaps an image". In that image there are typically about 40 different collisions. Correlating tracks to vertices is not that easy given these conditions and is further complicated by the fact that particles can decay mid flight, causing their track to suddenly change. It would be great if we could be absolutely confident about the physical quantities associated with all of the particles produced in a collision, that's all we would need to know and this is essentially what's produced in high-energy physics simulations. However, reality is a lot messier. In addition to what I previously mentioned, the interactions between the detector material and the particles themselves also creates a lot of noise in the system. All of these uncertainties make it infeasible to compress an event down to just the reconstructed physical quantities, there is too much uncertainty and I don't think it will ever be overcome.
Lastly, I should mention that the energy densities reached in the immediate vicsinity of the collision zones is high enough to destroy any material known to man. The center of heavy-ion collisions reach a temperature of over a trillion degrees. That's 100,000 times hotter than the center of the Sun. No instrument could ever be placed there to make an accurate reading. The best we can do is place detectors a decent distance away (IIRC it's about a few cm) and try to catch what comes out.
[0] http://cms.web.cern.ch/sites/cms.web.cern.ch/files/styles/la...
I haven't spoken with him in some years, and I won't mention names, but a good friend worked on calibration of the CMS detectors (scintillation). The last time I did speak with him, that was a very interesting 2 or 3 hours. Although I don't recall us discussing the data processing aspects in any depth; my limited impression of those I mostly gained from random articles and whatnot.
P.S. I guess I'd bettter RTFA, now. ;-)