Every 7.8μs your computer’s memory has a hiccup

> Basically permanent magnets.

Basically iron, which can be magnetized or demagnetized just like you can use a magnet to magnetize a steel needle.

Since I've been maintaining core memory recently, let me add a few comments. Core memory is built from ferrite rings called cores, which can be magnetized in two different directions. By passing a current through a core one way, it can be magnetized the matching direction, and passing a current the other way magnetizes the core the other direction.

The important property for reading a core is that when a core flips from one state to the other, the changing magnetic field will induce a voltage on the sense line threaded through all the cores. But if the core stays the same, you don't get a voltage. Thus, by flipping a core to 0, you can detect if the core was in the 1 state previously. This allows you to read a core, but the read is destructive since the core is now 0.

Another key property of the cores is hysteresis. A small current through a core has no effect on it, but above a threshold the current can flip the core. This is very important because it means that you can put cores in a grid, and energize one X line and one Y line. Only where the two lines pass through the same core will the core receive enough current to potentially flip. This coincident-core technique is what made core memory practical. This grid of cores is called a core plane. For a 16-bit computer, you stack up 16 core planes and read or write them in parallel.

Finally, to write a core, you first access it to ensure it is flipped to 0. Next you access it with currents in the opposite direction, flipping it back to 1. But what about the bits in the word that you don't want flipped to 1? The trick is the inhibit line, which passes through all the cores in the a, in the opposite direction. If you put a current through the inhibit line in a plane at the same time as the write, the currents partially cancel out, and the core doesn't flip, so it stays as 0.

Thus, each core typically has 4 wires: X and Y wires in a grid to select the core, a sense line that passes through all the cores in a plane to read the value, and an inhibit line that passes through all the cores in a plane to inhibit the writing of a 1. Some systems combine the sense and inhibit lines into one line, but that increases noise. (You want the sense line to switch directions so noise from the X and Y lines cancels out (that's why it's usually diagonal). But the inhibit line is forced to run a particular direction so the current cancels out.)

If this isn't more than you ever wanted to know about core, I've written about the core memory in the IBM 1401 in detail here: http://www.righto.com/2015/08/examining-core-memory-module-i...

That's interesting, makes me want to see the graph of power vs cost for extra error correcting data but with longer refresh cycles on the memory (for that suspend-to-RAM application).

This is relevant to the hack you can do when trying to retrieve secrets from ram from a live computer: pull the dimms out, blow some cold air on them and then stick them in another computer that's ready to dump the contents. I've heard its a good attack against full-drive encryption because the keys are sitting in plaintext in RAM. I've also heard it works for up to minutes after the computer has been turned off.

I have a few war stories about this from back in the day when I worked in hardware design: we made a chip that was supposed to have embedded SRAM but the cell design vs process was a bit off and it turned out to leak charge slowly, making it a DRAM. No problem : we modified the software to hit every row every so often which in effect implemented refresh. Also the converse problem: if you have a bug in your hardware that results in refresh being disabled it can be hard to detect because as noted above cells typically have a retention time on the order of 10's of seconds. Add in the effect of programs running, which causes accesses that as a side effect refresh the row targeted, and it can take a long time before you realize there is no refresh. The symptoms can be subtle and confusing, because for the most part memory _is_ being refreshed, just not in the way you expect.

Oh wow. Thank you. I've been wondering all week why the CNC machines at my work store their data and info on battery powered SRAM after getting a warning that i'll lose everything if I don't change said battery in a couple weeks.

I always assumed everything was written to disk and loaded into memory on startup but apparently not. Everything critical to the machine running is stored on this SRAM.

It actually makes a lot of sense now why the manufacturers did it this way. I was really confused why in this day and age I have to change an SRAM battery stashed somewhere deep in the computer inside the control panel.

Good servers do not need periodic SMIs, stuff like temperature sensors are handled by the BMC. Of course it depends on the benches and it can be a pain to find them, but people using the Linux realtime patch for example really need servers that do but abuse SMIs.

Periodic SMIs in theory should only be used for stupid stuff like emulating a PS/2 keyboard and mouse from a USB one, which are safe to disable. SMIs are used to handle machine checks, access to persistent storage (including UEFI variables), and various RAS (reliability/availability/serviceability) features.

So long as the overhead is predictable (which is an interesting assumption) then it'll just contribute noise and not affect the frequency analysis.

clock_gettime overhead is on the order of ~20 ns (or about 60 cycles) on most systems implementing VDSO gettime, and is quite stable. The author is trying to measure something on the order of 75 ns.

Taking an average is the wrong technique here. The author is trying to measure variance between equivalent code due to hardware events. I mean, on average, DRAM access is very fast! But sometimes it's not, and that's what the article is about.

And in any case on a modern x86 system that call (which is in the VDSO, it's not a syscall) just reads the TSC and scales the result. That's going to be reliable and predictable, and much faster than the DRAM refresh excursions being measured.

Well, you can parallelize it by adding more sticks of RAM.

I don’t know modern RAM chip organization, but in the old days the bit cells were layed out in a square and an entire row would refresh in parallel. A 1Mb chip would then refresh 1Kb at a time taking 1K refresh cycles for the entire chip.

Maybe one of the reasons it's hard to parallelize is that the refresh operation requires electricity, and there's obviously a limit on how much of that it can use.

DRAM beats SRAM at density, power, and cost by pushing the refresh circuitry out of the NM memory cells and into the periphery of the NM cell matrix. There are only M amplifiers per NM matrix, so you can only refresh M cells at a time and must perform N refreshes per refresh interval to catch every row. Could you put in NM amplifiers to refresh all the cells at once? Sure, but then we would call it SRAM :-)

> It implies that the length of refresh operation is roughly linear in the number of bits being refreshed.

I think you are misreading it. The point to avoid locking all the memory each time, instead you only lock (via refresh) a small potion of memory each time.

Well, you can't really read from multiple addresses simultaneously.

> Why is it impossible to parallelize this?

One of the mentioned articles touches on this: http://utaharch.blogspot.com/2013/11/a-dram-refresh-tutorial...

> Upon receiving a refresh command, the DRAM chips enter a refresh mode that has been carefully designed to perform the maximum amount of cell refresh in as little time as possible. During this time, the current carrying capabilities of the power delivery network and the charge pumps are stretched to the limit.

I guess: it's actually hard to deliver power to refresh all the bits at once? Also note that ancient cpus like Z80 had the memory refresh machinery built into CPU as opposed to memory https://en.wikipedia.org/wiki/Memory_refresh

> What's the cause of the 3200ns+ readings?

No idea. Random noise? Timing interrupt? Some peripheral doing DMA transfer? Kernel context switch? System Management Mode?

https://en.wikipedia.org/wiki/System_Management_Mode

Feel encouraged to run the code and try to debug it!

Maybe the kernel timer tick?

> Why is it impossible to parallelize this?

Within a single chip, it is impossible by design... not by any physical nature.

The DDR4 spec has 16-banks (organized into 4-bank groups), which could theoretically refresh individually. But that's not how the spec was written: a Refresh command will cause all 16-banks to start refreshing at the same time.

However, it is possible to "parallelize" this rather easily: that's why you have TWO sticks of RAM per channel. While one stick of RAM is going through a Refresh cycle, the other stick of RAM is still fully available for use.

My assumption is that it is a better protocol to issue all banks to refresh at the same time. Otherwise, you'd need to send 16x the Refresh commands (one for each of the 16-banks). At that point, most of your messages would be "Refresh" instead of RAS / CAS (open row, or open-column) commands, needed to read/write data.

If you really want parallelism, get multi-rank RAM or stick more RAM per channel. But even then, if the two sticks of RAM refreshed at the same time, you'd have fewer "Refresh" commands in general. So it still might make more sense for memory controllers to keep sticks of RAM all in sync with regards to Refreshes.

The Goertzel algorithm is O(N) for a single frequency component, which beats the FFT’s O(N log N), where N is the length of the data. The FFT gives you the whole spectrum though, whereas Goertzel “costs” O(N) each time. This might matter if the peak is not exactly where you imagine (e.g. 7.8 instead of 8). Furthermore, there’s a been a ton of work put into optimizing FFTs for a wide range of platforms, special cases, etc, so the FFT might actually run faster.

Further reading: http://www.fftw.org/pruned.html

Watch out, there is a bit of cargo cult there. Particularly around the operations preparing the data for the FFT.

The C part is rather straightforward. The pinning is helpful, but not really needed. The MTRR was a dead end (but I left the code since it is fascinating). ASLR is kinda needed since it destroys determinism between runs.

With all these tricks I still struggle to get the code running on Kaby Lake. It seems more stable on older CPU generations.

DDR4 btw has a few major differences:

1. Banks are now split into Bank Groups. Typically 4-bank groups, with 4-banks per group. That's 16-banks total.

2. While waiting for one bank to finish a command (ex: a RAS command), DDR4 RAM allows you to issue commands quickly to other groups, usually in 2-cycles or less. In effect, Bank Group 0 can work in parallel with Group1, Group2, and Group 3.

3. The 4-banks within Bank Group 0 can work in parallel, but at a much slower rate.

4. This allows DDR4 RAM to issue 16-commands in parallel, one to each bank group, as long as certain rules are followed. DDR3 RAM only had 8-banks. This allows DDR4 to be clocked much higher.

5. DDR4 uses less energy than DDR3.

-------

That's about it actually. So the vast, vast majority of the information in that document remains relevant today.

Modern graphics systems are too flexible for this - you can reprogram the graphics memory layout in such a way that would break the refreshing. But this technique was used on early PCs (with CGA graphics) and many of the 80s microcomputers.

It's just a conversion factor to make the units come out nicely. He could have easily done T = 1/f to give the refresh interval in seconds, but with such a short duration, it helps to convert it to nanoseconds (1e9 ns = 1 s).

It's a blog post about something more or less unrelated to CloudFlare that just happens to be on their blog. This comment is lacking in any real substance other than snark.

Have you tried reaching out to them to try and resolve the issue?