Neutrinos Faster Than Light, or FPGA artifacts? | Hacker News Reader

Neutrinos Faster Than Light, or FPGA artifacts?

blog.zorinaq.com

223 pointsmrb57 comments

A nit, but this struck me as highly speculative. I, like the author, want to see the FGPA equipment analyzed further.

I kinda felt like I was led down the garden path a bit here. In the end, the entire essay (to me) boils down to "Stuff we don't know about could be stuff we should worry about" This is a true statement, but I'm not sure how useful it is to the reader.

Perhaps it would have been better to list all of the possible problems, then show why the FGPA was the biggest concern. That way I would be more informed about what the state of the discussion is. While it's really cool the author can outline the myriad ways an FPGA device reading can be misread, context here is more important than the ability to impress us with all the technical details (which are also important, no doubt, but only with context) Put another way, this was a bit of "nerd porn" -- lots of cool little technical details that need to be considered when dealing with FPGAs in places like this. For that, it's definitely great HN material.

Just a structural criticism of the essay, not the topic or author. The topic and writing quality were awesome.

I've worked with high resolution timing devices, data acquisition systems and NIST traceable timestamps, PMTs and APDs and FADCs. 10ns precision is a relative cakewalk, and can be done with about $600 in equipment. Often times some people neglect some latency introduced due to analog filtering, but it's nowhere near the order of 60ns. Furthermore, most of these systems are almost always calibrated daily with fiber optics, lasers, or other subatomic particles or particle accelerators. So often times the delay is actually relative to a particle (usually a photon) moving at or near the speed of light in a medium with a known refractive index. I'm sure it's not too hard to find the whitepaper for the equipment if you really wanted to look for it.

I'm not saying it's not possible, it's just extremely unlikely. The author is asking for FPGA analysis when he should be asking for calibration analysis.

I agree it is easy to design the system with 10 ns precision. Just like it is easy to calculate trajectories using different units, yet the $125 million dollar Mars Climate Orbiter space probe crashed due to a metric mixup that could have easily been avoided. Another way I look at it is that the OPERA experiment is either going to be concluded by brand new physics, or by a statement like "doh! I can't believe we screwed that simple thing". Out of all the systems they described, this one seems the more likely to be erroneous to me.

I understand that it's very possible the FPGA is the problem. However, it's a cakewalk to continuously monitor and calibrate FPGA latencies. In fact, it's probably done in conjunction with the calibration of APDs/PMTs. If they aren't doing this, then the whole experiment has much bigger problems. My point is that I personally feel it's incredibly wrong to single out the FPGA DAQ systems, as the discrepancy of 60ns is so incredibly large compared to what is easily attainable, and the delay is something that's always compensated for in calibration. I think that there's some other error in the system, and my gut feeling is that it's due either to geometry or calibration.

> I'm not saying it's not possible, it's just extremely unlikely.

Which is more unlikely though, that the FPGA has an issue where precision can be worse than 60ns, or that we're wrong about the speed of light / the ability of matter to travel beyond that speed? We're talking about hard to believe things here, so we shouldn't discard the possibility that this was the failure point simply because it's unbelievable.

I know you're not discarding the idea, and I welcome the context you bring to the discussion; I just wanted to point out that in this case there really isn't much that should be thrown out as impossible.

But saying it's extremely unlikely doesn't mean the speed of light is wrong, it means the error could be elsewhere in the system.

A systematic uncertainy is very possible, but a systematic uncertainty is a combination of things.

Assuming it's on the FPGA/DAQ side first and foremost is silly because there's multiple systems involved and they are geographically very far apart. Maybe someone forgot that they cut the cables shorter somewhere, or someone is using different cable with a different velocity factor than they are used to. Maybe their geometries are off by 18 meters.

The experimental setup and calibration, along with how it is performed, is much more important than any single component. Assuming it's the (seemingly) most complicated piece in the experiment is wrong. Everything needs to be checked and double checked at this point.

Still without the context, but those two FPGA mechanisms he suggests don't sound that compelling.

Why would they be using an FPGA that needs cached DRAM? It seems more likely that they're just using it for plain old logic. And you would assume that temperature-dependent changes affecting the crystal wouldn't be big enough and systemically consistent enough to account for the 60ns. But I guess we'll see.

Beats me.

The problem for the layman with this type of article is that the author can pick something that is unknown to us, dump a ton of speculation on the reader, and at the end you've got what, exactly? A big pile of technical speculation, that's all. Sort of a rehash of all the physics classes and texts that the author may have consumed. The ability to speculate on possible instrumentation problems does not correlate with the probability that the instrumentation in question is actually important.

Put another way, this article would be a great comment inside of a larger conversation about possible problems, but it's not so good as something that stands on its own. In the author's defense, perhaps it was written as part of a larger web conversation that he can then refer people to.

But it's not technical speculation - he pretty much says he knows nothing about these devices, then assumes they're not too dissimilar from a home PC. More just idle gossip.

How long before we can expect new results/analysis from Fermilab?

p.s Healthy discussion going on at ResearchGate.

http://www.researchgate.net/topic/Physics/post/What_do_you_t...

> Perhaps it would have been better to list all of the possible problems,

Here is one attempt;

http://motls.blogspot.com/2011/09/potential-mistakes-in-oper...

Firstly, if this FPGA-based system is using DRAM and implements caching, results may vary due to a single variable or data structured being in a cache line or not, which may or may not delay a code path by up to 10-100 ns (typical DRAM latency). This discrepancy may never be discovered in tests because the access patterns by which an FPGA (or CPU) decides to cache data are very dependent on the state of the system.

Clearly, the author did not know what he is talking about.

I'm not an expert on such systems. Could you to elaborate on this a bit more?

The author assumes that FPGA works in a way similar to CPU, and makes various assertion. But the way it works is that it implements a logic/circuitry except that such logic/circuitry was specified by some bits in the memory (hence the word "programmable"). The logic stays unchanged and all bits in the memory that specifies the logic needs to be available at any given time, so the clock cycle is rather deterministic and there is no such thing as "caching" the bits that describe the logic. Granted, FPGA does interface with other components such as external DRAM so technically it could be affected by some caching behavior there, but it is highly unlikely that caching plays as big of a role as would be if it is run by a CPU.

The only design I can think of in an FPGA that would use a "DRAM cache" is a processor. And by definition, no one assumes processor code execution to be timing accurate to 10ns.

Lot of people use SDRAM with FPGAs. http://www.google.com/search?channel=fs&q=fpga+sdram&... SDRAM is used commonly with FPGAs for storing largish amounts of data at a reasonable price (static RAM is fast, predictable, and very expensive for large fast memory arrays).

SDRAMs are horribly complex state machine driven pipelined devices. On a system I was working on, a simple write operation to the SDRAM could vary from several clock cycles to 52 (IIRC) clock cycles, depending on what was going on before the "simple" write. I forgot what the worst case stack up was, but it was probably something about a refresh followed by a page miss that needed a precharge...

AnandTech has a good intro... note the simplified state machine diagram (Figure 2) is not simple. If they published the full state machine, your head would explode. http://www.anandtech.com/show/3851/everything-you-always-wan...

Trivia: The data throughput listed by SDRAM manufacturers is the best case sequential access fully pipelined scenario when all -nine- eight planets align.

Speculating here: The variability of SDRAM access latency is one reason for putting a high speed cache in front of it. For instance, the cache could be a simple FIFO operation to buffer the data to avoid dropping data when an operation got hit by a long latency stackup. The SDRAM may have the average throughput to handle the data stream, but the (unpredictable) major bumps in latency may required "cache" buffering to keep from dropping data.

An FPGA is not a magic, nondeterministic device, but nothing more or less than a lot of ordinary logic gates in one convenient package, implemented with bundles of small LUTs and flip-flops. If you don't use your FPGA to implement a memory controller or other device with variable latency, you don't get a device with variable latency.

Timing instruments can be challenging to design, but as a rule, they are fairly trivial to test (waves hands dismissively). I wouldn't look for the problem at the gate or RTL level in the timing gear... but the interface between the insanely complex detector hardware and the timing gear is harder to dismiss with hand waving. If there's a bug, it will be in some part of the system that can't be easily characterized for its so-called "residual noise" performance.

It is pretty common to use DRAM with an FPGA to manipulate large amounts of data. I clarified my essay: "eg. to store or manipulate large quantities of timestamps or events data that do not fit in SRAM".

If so, I suggest you change the title to "Neutrinos Faster Than Light, or DRAM artifacts?"

Changed to "artifacts of the FPGA" (as in "the FPGA platform", which includes the crystal oscillator, DRAM, and everything else that might be responsible).

I understand you don't want to change because it does invalidate a big portion of your article. But by dancing around the problem, instead of addressing it, you did weaken the trustworthiness of your article, and make it sound more of a witch hunt (read: chasing idea without much evidence).

You have two points that I think are reasonable to raise: (1) is there a mistake (intentional or not) in the assumed margin of error in the measuring equipment? (2) is the DRAM caching behavior taken into consideration? Neither of these two points have anything to do with FPGA per se. Now by including the enchilada word "FPGA platform" into the fray (whereas it was clearly not the intention at the beginning), I feel like you are making things up as you go, which is unfortunate since part of your argument is valid.

I cannot title it "DRAM artifacts" because it would be only 1 of the 3 different causes I speculate on (DRAM, oscillator, FPGA clock).

I chose the word "FPGA" to match the terminology of the paper which uses the word as a synonym for the Data Acquisition System.

I chose the word "FPGA" to match the terminology of the paper which uses the word as a synonym for the Data Acquisition System.

If so, that's a problem with the paper in itself. (I haven't read it.)

It is pretty common to use DRAM with an FPGA to manipulate large amounts of data

True, but your essay confuses acquisition with storage. As a rule, by the time the data is stuffed into a DRAM somewhere, it's already been timestamped. If you're right, and it hasn't been, then, yep, that's (a) problem.

That's the beauty of revealing lots and lots of details about the measurement process: some "random guy on the internet"[1] can point out possible mistakes, or at least suggest where to search.

So some people from the telecom or HPC community who have much experience with lags and timing jitters can contribute their experitse.

[1] I don't mean to disrespect the author, just point out that he's probably not involved with the experiment at all.

>latencies of the order of 10-100 ns unexpectedly added or subtracted to a baseline

Perhaps my stats is a bit rusty, but if its the discrepancy is +60ns (only plus) & <20ns uncertainty, doesn't that rule out a -+100ns source (plus minus) as the cause of the (possible) error? i.e. We should be looking for an issue that shifts the results, not something that is adding noise.

I think that the author's point is that these latencies would, in fact, add a systematic bias to the measurements (i.e. they would shift the results, not add noise). I interpreted "added or subtracted" to mean that this bias could either be added at the source end or subtracted at the receiving end.

The diagram being discussed is labeled "LNGS" so I'm pretty sure it both green and blue relates to one end (receiving), just different parts of the receiving end.

"I tried that! Don't you think I would have tried that?" --Paul Richter, WarGames

Not that it's not a worthy thing to consider (after all, the claim is extraordinary), but it was probably considered.

The bug is almost always something you overlooked as too simple or obvious to check.

The observations are worth looking at. I could pull down a part of an article about CERN's 'faster than light' claim. Here it is; "An American experiment involving Fermilab and a Minnesota mine showed the same thing back in 2007, but the results were within a margin of error that kept anyone from jumping up and down about it. (The CERN results are within a margin of statistical certainty that, were this not such an unexpected result, it would be considered a new discovery.) Now the team plans to update that experiment with about 10 times more data". We can't rule out the possibility of an entirely new perspective on quantum physics. The tests are on and we can here a follow up story soon.

> Now the team plans to update that experiment with about 10 times more data

FYI, more data taken by the same experimental set-up isn't going to help. This is already a 6-sigma result, which means that a statistical fluctuation is ruled out as an explanation (~ 1 in a million chance). Instead, there's almost certainly a systematic error which is being made in the same way for every data point, so taking more data doesn't do anything.

6 sigma is 1 in 506 million.

Ahh, thanks. I just typed "six sigma" into google and pulled the number off the wikipedia page about that business practice. Should have gone with Mathematica.

Regardless of the technical details, I believe the author of the article is making the wrong assumptions.

He is talking about errors depending on the internal state of FPGA, or "the guy operating the device leaving the door open" leading to a change in temperature (very unlikely, but still you have to consider everything).

The problem is that all of these possibilities would lead to different readings on multiple trials, while we are not talking about a "one off" experiment, but one that has been tried several number of times and the result has been consistent.

An error might still be there, but clearly is not based on a random internal state or outside conditions, but must be systematic.

This is easy to figure out. Move the detection station to, say, half the current distance. Do the neutrinos still come out 60ns faster? Then it's a problem with the apparatus. Do they come out only 30ns faster? Then some of the Physics needs to be rewritten.

This sounds easy, but it's not.

The neutrino detector needs to be far underground (to shield it from cosmic rays), is pretty heavy (I remember something about 60 metric tons, but it could be more), its location needs to known very precisely, and it needs the same sophisticated timing infrastructure as OPERA has.

I guess it would be easier to build another neutrino emitter at a different particle accelerator, and direct the beam towards the OPERA lab (or another existing lab).

It sounds like most of the things that concern the author would introduce substantial amounts of scatter on top of some possible systematic shift, so I have a hard time believing they wouldn't have seen that.

They said that they measured the FPGA delay by shining a beam on it and seeing how long it took to register. It seems like there is a lot of room for variability in this practically un-analyzed system.

I don't want to sound elitist, but I really think the team double-checked all the trivial reasons which could explain this extraordinary result (à savoir : faster than light particles).

Probably, but think of all those times when you checked and double-checked your code, only to find out that it was a simple typo causing the bug.

This will need to be experimentally verified (preferably with a completely different set of equipment), but it doesn't hurt to double-check and triple-check the simple stuff.

Couldn't this possibility be eliminated by calibration?

e.g. having a source of neutrinos 1 km away from the receiver, and see if they still detect it 60 ms "before" (perhaps even before they fired them)

Is the cost/technology for doing this kind of quick check prohibitive?

I was wondering if the neutrino detector does have to be directed precisely towards cern or it's omnidirectional. Does anyone know?

A neutrino detector is omnidirectional. It would be almost impossible to make it directional even if you wanted to.

You can try to measure which direction the scattered particles from a neutrino collision travel. But the actual detector is omnidirectional.

But all that said, a strong neutrino source isn't easy to just do.

assuming the emission is also omnidirectional the required strength decreases by square of distance, so it shouldn't require a strong source for 1km.

I guess it's more difficult to tell exactly when the bursts are being emitted.

No, the emission is directional.

http://xkcd.com/955/ applies here.

Seems speculative. Haven't they measured the speed of EM, or possibly some other subatomic particle than the neutrino, with the very same equipment, and gotten the expected and scientifically agreed-upon speed as a result? It seems to me that testing the measuring/timing hardware with a known phenomenon would be a fundamental step before releasing and using it.

As I recall, the path taken by the neutrinos in this experiment is several hundred miles of solid rock, so, no, there really isn't any other type of particle that could be tested with it.

Sure there is... dark matter particles; now if only we can find them!

Aren't neutrinos part of the dark matter?

No.

That depends on how you define dark matter. (i.e. do you mean only cold dark matter).

Dark matter is just matter that isn't visible electromagnetically and neutrinos do qualify.

While that is technically true, the context of this comment should make it clear that I'm referring to cold dark matter, otherwise my comment about finding them wouldn't make sense.

But if this discovery is true then there is a lot we don't know about neutrinos. For all we know they could be dark matter, and there is no other particle.

For example if they are a type tachyon, then the closer to c they are the less energy they contain, this would actually make them very cold.

(Personally I doubt they do exceed c, but I do think that they are actually dark matter, only created via some process that keeps them slow.)

More info on the time-transfer system that was just released today: http://www.ohwr.org/projects/cngs-time-transfer/wiki/Wiki

Reply on news.ycombinator.com