Hacker News Reader

Comment by GauntletWizard | Hacker News Reader

The memory can recover one, detect two, and will throw up loud warning bells to any competent admin as long as it is doing so. If you see a sudden spike of ECC errors from a server, the memory is probably bad, and you replace it. If you see a sudden spike of ECC errors from a large number of servers, something is seriously wrong, and you investigate - I'd be more likely to blame it on bad power myself, or a EM Field/radiation source.

Even when trying to trigger rowhammer, the graph of bit flips is going to look like a logarithmic curve - Single-bit flips will be common, two-bits will be rare, and undetected flips will be real but ultimately not much to talk about compared to the discovered flips. You know they're there by the unrecoverable rate, and you replace the ram, discard any replicated data on that machine, and move on.

View on HN

The memory can recover one, detect two, and will throw up loud warning bells to any competent admin as long as it is doing so.

I don't have personal experience here, but one of the important claims in the paper is that this warning is not given on all servers:

  Unfortunately, server vendors routinely use a technique   
  called ECC threshold or the 'leaky bucket' algorithm where 
  they count ECC errors for a period of time and report them 
  only if they reach certain levels of failure. From what we 
  understand, this threshold is commonly above 100 per hour, 
  but this remains a trade secret and varies based on the 
  server vendor. So, to see ECC errors (MCE in Linux or
  WHEA in Windows), there generally needs to be 100 bit flips 
  per hour or greater. This makes “seeing” Rowhammer on 
  server error logs more difficult.

  In addition, we have observed some server vendors will 
  NEVER report ECC events back to the OS,although they might 
  get logged into IPMI. Typically, users expect to see 
  correctable ECC errors logged directly to the OS or that 
  halt the system when they cannot be corrected. During our 
  investigation into this phenomenon, we even encountered one 
  server that neither reported ECC events to the OS nor
  halted when bit flips were not correctable. The end result 
  was data corruption at the application level.
  This is something, in our opinion, that should never happen 
  on an ECC protected server system.

http://www.thirdio.com/rowhammer.pdf

On the servers that I use the server will log to ipmi sel only after a threshold but the Linux OS will get an MCE for each error so a monitoring tool can detect and alert even before the BIOS logs anything.

I really wish they'd have expanded on that somewhere and actually stated what manufacturers/models of server they tested and which ones don't report ECC errors.

Really, without details like that it's just unverifiable FUD.

Reply on news.ycombinator.com