Malloc Geiger Counter | Hacker News Reader

Malloc Geiger Counter

github.com

238 pointsmkeeter81 comments

Here I was hoping this was something about repurposing RAM as an actual geiger counter by measuring radiation-induced bit flips.

Actually me too. I did write a program a while back to investigate the rumors about this though. Spoiler alert: Doesn't really happen. (https://github.com/s9w/MCC)

I’m mildly disappointed aswell.

Well, I'm also happy this doesn't happen! RAM is designed to be stable!

If you have the resources, have you considered getting a source of radiation and putting it by the RAM? In a plane for example, background radiation can be 30x or so. It might also reduce the time to see a flip...

Other considerations:

1. The act of checking the RAM (I believe) refreshes the memory cells.

2. I suspect the act of regularly changing the RAM bits to have an effect on the data stored. I think when the bit transitions from a 0 to 1 or vice versa, some inconvenient piece of solar radiation could bias it to a different state. That said, you would have to separate that from accidental row-hammer events.

3. You would think that higher temperature RAM was more likely to bit flip (the Scientific term is more wiggly electrons).

4. RAM clock speed. The higher it is, the less time it has to stabilize at the transistor level.

5. Memory cell size. The smaller it is, the more likely it will be affected.

There are probably tonnes of variables, but these are the ones I can think of for now.

That's actually a couple of interesting ideas. I don't think however that 1) is completely correct? As far as I remember from my classes, RAMs are refreshed all the time on very small time scales exactly because they're volatile. Therefore a manual read shouldn't influence things.

Temperatures and clock speed would be very interesting. Not sure I want to even google for radiation sources though :D

DRAM reads are destructive, so reading a row always refreshes it. On the other hand, every row is refreshed every ~64 ms. (This is governed by tREFI, the interval between successive rows being refreshed. For example a module with 8K rows would require a tREFI of 64 ms / 8K = 7.8 us = 12480 clocks (7.8 us / (1600 MHz)^-1)). Increasing tREFI is one of those things people do to improve memory performance, because less time spent refreshing means more time available for useful work. Some memory chips can use very high values, which would be equivalent to ~200 ms, others corrupt data consistently much before that.

So while rows are refreshed very often, a specific row is only refreshed ten times per second or so.

Interesting, thanks for the numbers. That would mean that it would probably be feasible to significantly influence the mean refresh times.

Bit flips in RAM happen often enough to be an issue.

I make a desktop program with a sizeable installation base and it uses a form paged on-disk storage for larger data sets. The code has a fair amount of asserts in it and one specific assert would keep failing in a very tiny percentage of the cases. I kept reviewing the code and scratching my head before finally adding some extra logging for this case.

Lo and behold, in all failed cases an argument passed to a function had one of the bits set for no apparent reason. I.e. the offsets of pages in the cache would be:

    00000000000000 ...
    00000000100000 ...
    00000000200000 ...
    00000000300000 ...

But the args would come in as

    80000000100000
    00040000200000
    01000000300000
    ...

Followed up with the people who reported these, asked to run an overnight RAM test and ALL of them found issues with their RAM. That was quite an eye-opener.

You weren’t explicitly saying it, but some might infer it, so I think it’s worth mentioning that bit flips in RAM aren’t necessarily radiation-induced. Some other causes are heat and aging (both increase capacitor leakage, and DRAM cells are tiny capacitors), or lack of robustness in refresh logic.

Another cause, at least in the old days, was bad board design, in which badly-placed traces could induce currents in other traces. Of course this only happened two+ hours into the test.

I remember a blog post by Riot (league of legends developer) also about this happening. But I couldn't find it again when I was searching for it. I can't imagine the havoc that regular bit flips must cause.

One of the more interesting ways to test this is to register a one character typo of a very popular domain where the typo is one bit off.

Like, say, dacebook.com or facebooo.com

Spin up a web server, and you'll get traffic, but typically only one of the requests from a specific client.

it has to be a one-bit-error byte change in particular.

https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...

Were all the machines which reported that in the same building (your software could been used in house). If so, better check for radiation levels, for health risks; I have never seen a RAM test fail actually (apart from a completely broken chip)

As mentioned below it can deteriorate over time. A while ago I had a RAM stick fail. I noticed because GCC started segfaulting randomly and I haven't upgraded GCC recently. Sure enough a memtest revealed the fault. I think it didn't have enough cooling.

I only purchased RAM with heatsinks since then, and I didn't have such problems anymore (also my latest case has much better airflow)

I unplugged the wrong power cable once, and thereafter the RAID controller on that machine would report degradation after the system was running for a while. Finally ran a RAM error check, one bit in most of a stick was flipped. The last stick. So once the machine was all spooled up for a while, I’d get errors.

This is also how I learned 3ware was using RAM instead of its own memory. What is he point of hardware RAID if main memory can screw it up?

I have also heard of people swapping hardware on a defective machine only to learn that the power supply was bad, sending noise or brown power to the system. Swap in a new power supply and the components start to behave themselves.

If you've got a system where you can tweak the ram timings and/or voltages (most desktops will let you do at least some of this), it's pretty easy to induce failures, if you haven't run into bad ram naturally.

I've run enough systems with ECC ram to say the vast majority of ram works and will continue to work, but over time, a small amount will fail, again the vast majority is a single bit in the whole chip, but some will do multiple bits near each other (uncorrectable ecc error usually halts the system), or man single bit errors all over (many machine check errors per second makes the system go so slow, halting would be better).

I read a blog article years back by a game author who inserted CPU/RAM/Math sanity checks into the code (I forget what the reason he did this for was), and he was suprised with how often it detected hardware errors in customer's PCs.

Wish I could find the article again, I forget the details but it was a fun read.

but did you put a radiation source next to your non-ecc ram?

Probably better to use a CCD with electrical tape over the lens. I'm curious to see the difference in sensitivity between RAM and a CCD. I would think RAM is pretty resistant to bit flips or measures to counteract it, like ECC, would be more common.

This sounds like it could be a cheap, software-based alternative to expensive radiometric hardware, interesting!

At CERN I used to work on radiation measurement and characterization using ram chips (two different types with different sensitivities to neutron energies), but also characterisation with a ccd. With the ccd you can distinguish the type of the particle based on the track that it leaves behind, but the problem is the very low cross section (area, sensitivity) of the chip.

To get a track the trajectory has to be in the CCD plane? Doesn’t that mean abysmal efficiency?

You can also get a trajectory with a stack of CCDs where the trajectory is not in the plane of the CCD. Actually the stack may be very high with quite some distance between individual CCDs, because you need trajectory in and out of detector elements like calorimeters that are a few meters in size. Also, trajectory may be bent because of the particle's charge and a magnetic field, for that you need a large distance to get the "amount of bend" if it is small.

But yes, there always is an efficiency, precision and identification tradeoff.

Hello, colleague!

For folks who might be interested in more details on this: The _cross section_ allows you to quantify the frequency of a specific event occurring in your chip when bombarded by a specific particle (or class of particles). For example, in a case of an SRAM, you might want to calculate the cross-section for bit flips due to 24-GeV protons. The result that you get will be in units of cm^2; you can also interpret it as the physical surface area of the chip, multiplied by the probability (0..1) that any 1 particle passing through will trigger an event. Let's say you measure a cross section of 0.01 mm^2. Then if you expose that component to a flux of 10^3 particles/cm^2/sec, you will get an average of 1 event every 10 seconds.

The cross-section is only valid for one combination of the (event type, particle, energy) tuple. But often in dosimetry, errors on the order of 20% are acceptable, so we can "abuse" the results a bit. Within one manufacturing lot of chips, the cross-sections should be reasonably consistent. Typically you would estimate the mean and variance by measuring a few samples from the lot.

Then, to accurately measure a radiation field using these findings, you would generally need to measure the flux of every particle-energy combination separately. This is impractical, and for SEE* concerns we generally get away with distinguishing thermal neutrons and high-energy hadrons (which are separated by several orders of magnitude in energy). For this, it is sufficient to have 2 types of SRAM where each has a dominant cross-section for one of the particle classes of interest.

To measure the type of radiation that causes long-term gradual damage (TID) such as X-ray and gamma rays, you would use a different principle of measurement altogether (RadFET, P-i-N diode, floating-gate dosimeter...)

Radiation measurement can be fun!

[*] single-event effects, including bit flips and latch-ups

Hello fellow RME comrade :)

I worked on CCD and Later a CMOS chips for particle detection for the ATLAS detector during my undergrad! Well, on FPGA test controllers for them. I think They solved the cross section problem by having lots of layers of these chips in the detector iirc.

A few months ago I ran across this article about doing exactly that (well, CMOS rather than CCD): https://phys.org/news/2014-06-smartphone-detector-app-positi...

Apparently it works reasonably well.

I'm not sure if it's still active but there was a project to detect cosmic rays using smartphone cameras flipped upside down or with tape on the lens: https://crayfis.io/

I was ready to give it a go ona few devices laying around, but the Google Groups signup story is a turnoff.

Indeed. Looks like it's been abandoned and stuck in beta

One of my favorite programming stories : http://archive.is/O70Rg

According to HN user idoh, it's indeed possible, and it's used in serious military systems.

https://news.ycombinator.com/item?id=3179852

> So, my mom is a former Maxwell employee who now does her own defense contracting, and makes this same type of part.

> As I understand it, when there is a nuclear event, it generates x-rays followed by the EMP. The goal is to have warheads in flight to be able to continue to their target, so the strategy is to employ an NED. When an event is detected, the warhead shuts down its electronics for the duration of the EMP, and then powers back up.

> With her NED, she uses an ASIC for detection. I might get some details wrong, but the ASIC has a physical array in it, and the x-rays flip bits in the array. When enough bits get flipped, one can infer a nuclear event. Because it is an ASIC it is really small, which has important advantages for space-born avionics.

Thanks, Кomrade, we now know that electronics of latest americaН nukes can be dizabЛed by x-ray emissИons.

I had exactly the same thought.

same here

Sometimes I miss being able to hear the hard disk read heads moving around. It was a surprisingly accurate way to tell what your computer was up to. Someone must have made a simulation of this, right?

I was just thinking about this. Other examples that come to mind are reading CD/DVDs, dial up modems, and turning on old CRT monitors

There's an anecdote about Jim Gray: he was once asked by a company to help troubleshoot a performance issue in their database (this was in the 80s or 90s I guess). Surprisingly, he did not start by looking at any code, but went directly to the server room and listened carefully for a while. Then, he said what the problem was. (IIRC it was an issue with wrong/missing indexes.) They went to look at the code, and it turned out he was right. Everyone was astonished, how he did this. He was "merely" listening to the sound of the spinning disks while the problematic queries were running.

Whether these stories are true or false I love them.

Ok, I found the reference for this story [1]! It turns out I messed up some details, but the core of the story is true. (It was not a company but Alex Szalay [2] at JHU, and it was not an indexing but a layout issue.)

Jim asked about our "20 queries," his incisive way of learning about an application, as a deceptively simple way to jump-start a dialogue between him (a database expert) and me (an astronomer or any scientist). Jim said, "Give me your 20 most important questions you would like to ask of your data system and I will design the system for you. " It was amazing to watch how well this simple heuristic approach, combined with Jim's imagination, worked to produce quick results.

Jim then came to Baltimore to look over our computer room and within 30 seconds declared, with a grin, we had the wrong database layout. My colleagues and I were stunned. Jim explained later that he listened to the sounds the machines were making as they operated; the disks rattled too much, telling him there was too much random disk access. We began mapping SDSS database hardware requirements, projecting that in order to achieve acceptable performance with a 1TB data set we would need a GB/sec sequential read speed from the disks, translating to about 20 servers at the time. Jim was a firm believer in using "bricks," or the cheapest, simplest building blocks money could buy. We started experimenting with low-level disk IO on our inexpensive Dell servers, and our disks were soon much quieter and performing more efficiently.

[1] https://cacm.acm.org/magazines/2008/11/549-jim-gray-astronom...

[2] https://en.wikipedia.org/wiki/Alex_Szalay

Another classic you’ve probably already read but linking it in case you haven’t:

http://catb.org/jargon/html/magic-story.html

I remember being acutely sensitive to the sound of a good floppy being read vs. a bad floppy when I was getting into computers.

Likewise being able to hear if you and the BBS were properly connecting or not. Hearing your modem retry the handshake after dialling 90 times to get into a popular one was heartbreaking.

When playing with Agat, Soviet Apple II clone, I've altered couple of bytes in RWTS (reat/write track/sector) routine from nop;nop; to clicking sound. That way I got gentle rattle wheh floppy was reading/writing okay and pauses when something was going wrong.

Later I've found out that piece of code was written by Wozniak himself.

I have old spinny disks and it is a VERY, VERY simple and useful feedback for DB work to hear it fighting itself.

As an aside I get the feeling software is being written more and more to assume SSDs. Starting up visual studio from fresh on my machine will thrash the disk. Disk read is roughly (average) 2MB/sec throughout. Disk queue length will hit 10 several times. It takes 4 mins 50 secs (290 seconds) before the disk will calm down - I just timed it. Total mem for the process when all's finished - 204MB! That's all!

My feeling is the VS code is spawning many threads which are each fighting each other for the disk. It's really, really poor.

While Visual Studio is undoubtedly a great IDE, it's biggest issue for me always been performance, especially when paired with ReSharper - this has been the case even before SSDs were ubiquitous.

Honestly, I'd strongly recommend you get an SSD - prices are really cheap nowadays.

Is VS a great IDE. It's laggy, it's too 'helpful' eg. it's going to add a close bracket if you type an open bracket unless you tell it not to, in which case it's a bitch to customise (took me 20 mins to find that option) and no luck so far if I want to adjust eg. menu & tab font to make them larger.

It can do refactoring etc. and has context-sensitive help/completions but honestly I work in emacs because it offers so much more - except the high-level refactoring etc. Omnisharp works, sort of... VS just isn't as good as emacs in many, many basic ways.

Anyway, I'll look at SSDs and stop nattering about editors.

I do think it's a great IDE, apart from the lag - though that is obviously a big issue.

In fact, after working with VS for years and getting more and more annoyed by the lag, I tried JetBrains Rider - I'm a total convert, it's one of my favourite ever pieces of software! The UI takes a bit of getting used to if you've been using VS for a long time, but not that much. Plus there's an excellent dark theme that is quite close to VS's.

Rider is absolutely stuffed with features and config switches - it's close to feature parity with VS, with VS able to do a few things Rider can't, and vice-versa. Rider absolutely flies by comparison to VS, and is more stable too. Oh, and edit-and-continue "just works" in Rider, whereas I've always found it problematic in VS.

After getting an SSD, trying Rider is my next suggestion :)

I really don't like VS. I did use jetbrains and whoa! Nice!. I'll check out Rider, which is totally new to me (thanks).

The thing about "I do think [VS] a great IDE" is you haven't experienced emacs so you don't know what can be. What emacs is bad at: all the things VS is good at, the high-level code 'understanding'. But honestly for writing/modifying/moving around/various other stuff, emacs is just so astonishingly comfortable when you get used to it. It's not "I don't need to use the mouse", more "I barely need to think about it". And then you've got stackable clipboard, registers, macros from heaven, regexps that just work, so much more...

And perhaps best of all, it's written by programmers for programmers. VS feels like management have been involved in featuritis and "UI/UX experts" as they think of themselves but aren't, have been allowed out of their cage too often.

I'll not recommend you try emacs, it's a lot of investment but if you could get emacs to do high-level stuff as well as the low-level... Right, time for me to stop hijacking this thread!

> As an aside I get the feeling software is being written more and more to assume SSDs.

I agree with this observation in general. I don't use VS Code in particular but you've given solid evidence of a seek-heavy/HDD-unfriendly workload.

> My feeling is the VS code is spawning many threads which are each fighting each other for the disk. It's really, really poor.

I agree this is a poor match to your hardware, but if you're saying this is evidence of poor engineering, I disagree. IDEs' most demanding customers are using high-core-count CPUs with SSDs and expect top performance with heavy workloads. So the Visual Studio team needs to make this fast, and I don't think they have much reason to care about HDD-based installations' performance. Most software developers won't have trouble affording SSDs to store their OS files and current project, and the ones that will aren't paying to drive IDE development either.

This is one of my biggest pet-peeves. Whenever I have to help people debug performance issues with their machines it's almost always due to a bottleneck on their disk.

For my family members that have laptops with those shitty 5400rpm HDDs with 8MB read/write buffers - I can't blame them and just tell them to upgrade to an SSD. I've witnessed Windows update hogging 100% of the disk activity for 30+ minutes after reboot on these machines until they're actually usable...

But for my developer friends who I figure should know better, they never seem to think of looking into disk activity. It's almost always things like applications hanging/blocking on disk due to lots of different processes trying to read/write at the same time on an HDD or "small" RAM size causing constant thrashing on a disk due to memory being paged in and out of swap constantly.

I randomly saw a poster for this paper: http://www.ise.tu-berlin.de/fileadmin/fg308/publications/201... which implements something like it!

Demonstration: https://www.youtube.com/watch?v=gWJtGZOp3K0

https://github.com/dbermbach/audiocues

Scooped back in 2000, if not before:

> We created a network monitoring system, Peep, that replaces visual monitoring with a sonic `ecology' of natural sounds, where each kind of sound represents a specific kind of network event.

https://www.usenix.org/legacy/publications/library/proceedin...

The Vice emulator for the 1541 family of floppy drives has optional sound output. Throws you right back to the 80s!

You could reprogram these drives to play songs by vibrating the heads

This immediately came to my mind as well. Whenever the computer was growling, it was better to not mess with it and wait it out.

https://en.wikipedia.org/wiki/Resource_Monitor

I've been meaning to get around to basically plagiarizing this UI for a personal linux system monitoring tool.

I've tried a bunch of the popular linux disk/resource usage monitors but I find none are as flexible/nice to look at as Windows' Task Manager overview graphs and Resource Monitor. Any time I'm using a Windows system I will dedicate a monitor to just having Resource Monitor's disk usage tab open (although it can result in nontrivial CPU usage under certain disk access patterns).

Here's a similar thing I did once...

https://github.com/gordol/ld_preload-sounds

Generates WAV output by hooking malloc() and read().

You can totally hear the standard libraries loading in, I started this thing on a whim of silliness...

It started out as just 38 lines of code.

https://github.com/gordol/ld_preload-sounds/commit/2d51b3e7b...

I think projects like this, which try to give normally invisible qualities of software some corporeal existence, are really important. Obviously this is too course-grained for diagnosing specific problems, but in environments like our phones or our full desktop environments, being able to turn on feedback about what is taking resources and how has the potential to help us build general feelings about particular ecosystems.

I have been Linux only for 10 years, so I cannot try.

I have used similar tricks with gdb to play a sound when some high-level function is called, can be really useful. However, for malloc I would expect it to be called so often that a human cannot hear the difference. Or the program needs to measure the rate and select the tone according to the rate. Of course it depends on the program, but at least if there is enough allocation to be worried.

I love the idea of using sound as a mechanism for debugging code. It works wonderfully in other areas. I used to run a complex atomic physics experiment in graduate school which involved a series of steps repeated over and over again, with slight changes in the sequence every time. Over time I got really attuned to the sound of the machine: the clicking of shutters, hissing of valves, a annoying 10kHz note that was generated by an amplifier driving AC currents, etc. and could easily tell by ear if something was off (which happened a lot!). I would also use sound to tune control loops by sending the error signal to a speaker or earphones and listen to the noise. It complemented a signal analyzer wonderfully.

Can gdb be configured to somehow “hum” a program by converting a series of operations into sound and we can hear our programs in action? No idea how to go about it though, and how to convert operations happening at the CPU clock rate into the audible regime, but seems like it’s worth exploring.

This is actually a brilliant idea.

If you change the pitch by the allocated chunk size (maybe in log scale) or use different sound for free(), someone could be actually detecting memory leaks just by listening to the app.

I was thinking that it’s useful in games or other applications where you don’t want to malloc at arbitrary times (eg in games you might malloc once in startup or on level start and then reuse that memory instead). Many applications with real-time requirements often work similarly (eg high frequency trading), so this would be a fun check to make sure your application is well behaved.

I like the idea of changing pitch by size and also representing frees, but I’m not sure it would be useful to find memory leaks since there’s no real way of matching them up in any non trivial application.

When I worked at a major game developer, our custom allocation library handled this more directly: malloc would crash the program if called during normal gameplay.

Hah, that's certainly one way to do it!

People did this with network utilization decades ago: https://www.usenix.org/techsessionssummary/peep-network-aura...

The snoop command on SunOS and IRIX had it in the 90s too.

I loved the paper! The technique its authors present is more elaborated, though: they mix different sounds extracted from nature, such as birds and wind, and allow mixing different streams to create something pleasant to a sysadmin's ears. I liked this paragraph in special:

"Due to copyright limitations on existing natural sound collections, Prof. Couch has spent many hours with a Telinga parabolic nature microphone and Sony DAT or digital minidisc recorder in search of the perfect bird."

Thanks for sharing this paper!

While some say here that they are disappointed that this isnt abour radioactivity, for me it was the opposite. Wheb reading the headline I thought, oh no not another one of those, wondering though how you can get malloc in.

Read it, and I think this is actually a great new idea.

Back in the day sysadmins would just hooks up the least-significant address line to a speaker. That way they could monitor the mainframe aurally while reading and drinking coffee instead of watching the bank of flashing lights.

Here's a version for Go: https://github.com/lukechampine/geiger

What a strange, but interesting, idea!

The broader idea here is that there are probably an infinite number of creative ways -- to measure different aspects of what software does, and to instrument that output in different creative ways that can be experienced via human senses (in this case, via sound).

Thanks goodness this is not a renaming/rediscovery/attribution-hijacking of a well-known and old concept (reference count, allocation count, etc.), something every other person these days seems be engaged in.

This is really cool! It just occurred to me that the same approach would work to identify non-sequential reads from applications that access data from disk/tape. Thanks a lot for sharing this project with us :-)

well I'm sure as hell not using this on any Haskell* code (imagine we apply the malloc sound effect to generic allocation)

* Any GC'd language really, JS, Java w/generics etc

I think inefficiencies in these languages come more from garbage collection languages. Allocation is faster since it is optimized with the knowledge that allocations will be GC'ed, and static analysis can help a lot for the short term stuff. Memory can be allocated with linear allocators inside arenas for short term memory, and be moved to longer term arenas when it lives longer. Stuff like that.

> well I'm sure as hell not using this on any Haskell* code

I think this is literally already a feature in Haskell, except it beeps on GC rather than allocation.

https://downloads.haskell.org/~ghc/latest/docs/html/users_gu...

That's cool, but could you tell me how to use it?

When I try

  ghc -B hello.hs

I get an error

  Missing file: ./settings

If I then

  touch settings platformConstants

the error changes to

  Can't parse "./settings"

What am I supposed to put in this settings file? I'm but a Haskell noob, and this is the first time I've heard of it.

EDIT:

I found it! The correct command is

  ghc hello.hs +RTS -B

I can't seem to get it to beep on macOS, though.

It just prints \a - it's up to your Terminal what it does with that.

Does print "\a" work in your terminal?

Yes, it does, but my terminal's "silent bell" feature had been enabled. I disabled it, and it worked, though only while compiling, and not during runtime, strangely enough.

Reply on news.ycombinator.com