Do those quotation marks mean "enterprise labelled", such as the WD RE4-series? Are there good data on how they perform vs the consumer series? By good I mean like the data your company provided, samples of thousands of drives.
Those drives aren't designed for HDD farms though. They are meant to be used in environments where a failure is a big problem and paying a lot more for a bit more reliability is worth it. Like that 1U server you got in a datacenter with 2 disks in RAID1.
If you have thousands of disks replacing is part of the daily routine, and isn't an issue at all (as numbers suggest in article/comments)
I buy all enterprise drives. Not because I can't handle replacing a disk; I live within walking distance of the coresite santa clara location where most of my servers are, and actually kinda enjoy that sort of thing. (Yes, yes, I'm sick. But what of it?)
I pay double for 'enterprise grade' drives because more often than the consumer-grade drives, they fail clean.
That's the thing... what does it mean to have a drive "fail?" The vast majority of my failures with consumer grade drives just /degrade/ rather than outright failing. They get shittier and shittier over time. And yes, with sufficient software you can detect this and automatically fail them, but I don't have that. the "enterprise sata" stuff? More often than not, the things actually fail before they degrade to the point where I notice them causing problems with other things.
I buy enterprise grade, not because they last longer, (In fact, I see no evidence that they do) but because they tend to work or fail, whereas consumer grade drives exist on a continuum between "working" and "failing"
(Of course, even the enterprise stuff isn't 100%... but it's much better.)
Although I have seen some evidence of consumer grade drives "failing dirty". E.g. in 2002 I tried a couple of 5 year warranty Seagate Barracudas in a new machine; it didn't take long for one of them to outright fail a 4K portion of the disk. I actually wrote a little C program to recover everything but that one bit (and it was a file I could then recover the missing data from), and switched back to SCSI enterprise drives for my main system drives and haven't looked back (granted, those are as fast as I can buy, and that means SCSI enterprise). And that includes a couple of machines I built for a non-profit that are exactly what datphp describes.
And, yeah, these best of the best drives do fail, at least I had at least one of the Seagate Cheeta 10K drives I bought back in 2002 completely fail, I think both eventually, the 2nd after 5 years had passed.
'ddrescue' is a program I use in similar situations. (not to discourage you from writing your own; doing that sort of thing leads to a deeper understanding of what is going on.)
But, yeah, confirming exactly what was the problem was good, especially since it was extremely odd. All blocks readable except for those 8, although the drive knew it was in trouble.
It reminds me of problems others have reported, especially those using high level error checking file systems like ZFS, where the drive accidentally writes to the wrong location correct data correctly, so the internal CRC passes on reads. The firmware in drives is said to be getting to full OS complexity....