Comment by cpuguy83 | Hacker News Reader

Comment by cpuguy83 | Hacker News Reader

Parent Full thread

In another life (2007) I worked for a company that does digital signage. We had about 8000 systems deployed across the US. The software on them relied heavily on SQLite. At some point we started seeing random corrupted SQLite databases.

Trying to figure out what was causing this we realized our pc manufacturer was using the same sku but hardware inside could vary... and in particular the hard drive. Every system with a corrupt DB had a particular brand hard drive (WD), the ones that didn't fail was another (Seagate).

We never really spent much more time on this other than to lock down the hardware config and replace all the hard drives in the failed systems with the "good" brand.

Super interesting to see this. I'm assuming it was: Disk drives that do not honor sync requests

We were of course using consumer grade equipment, because the company was pretty cheap, but also we had so many to deploy. I wonder how much this cost the company to fully resolve. Between systems not playing, having to send technicians, shipping systems overnight, time troubleshooting, etc.

Reminds me of the Maxtor DiamondMax Plus 8 disk firmware problem. This was a model of hard drive back in 2004-2005 that had a known issue where it would corrupt it's own firmware. The disk would literally overwrite and corrupt portions of firmware code which would render the disk unusable. Heat seemed to exacerbate the problem, and the problem caused the disks to generate a lot of heat so it was self-perpetuating.

I worked at a hospital at the time, and we were upgrading from Win2k to WinXP, and replacing a lot of old HP Celeron 733 MHz systems with newer HP dc5000 AMD desktops. We deployed roughly 1,000 desktops over the course of a year and a half, and fully 25% of the systems that had a DiamondMax Plus 8 had disk failures within the first 6 months of deployment. Symptoms were always the same. System would get slower and slower until one day it would hard reboot itself and then claim there was no disk drive installed. BIOS would show the disk drive was present but that it had no capacity or name. Built-in BIOS self test would report a code 7 error.

It got bad enough that we refused to deploy systems that shipped with those drives. Our manager had to fight with the sales rep for about a month before they relented. We were ready to go to Dell just to get away from those stupid disks.

You can still find disk recovery sites talking about the issue: https://vitaldata.ca/article/n40p-data-recovery.html

Seagate had their problems too around that time.

The Barracuda 7200.10 was infamous not just for its bug but also for Seagate's response.

From what I recall, the SMART log wrapped around periodically - if the drive was reset when the pointer was at a certain value, it failed to boot with an assertion failure. 1 in 128 chance of a failure on a given power-up.

Thankfully someone found out that you could send a couple of debug commands to it with an FTDI cable and get your data back.

With how widespread both SQLite and WD are wouldn't half the world have apps crashing left and right, if just the hard drive brand was the cause of the problem?

The case of the disks not honoring fsync is famous. I learned about it from Brad Fitzpatrick, author of LiveJournal (later on the Go team at Google) writing about it in 2005: https://brad.livejournal.com/2116715.html

One of the first optimizations at basically every level is to not honor fsync requests. Network file systems lie. Operating systems lie about local volumes. RAID controllers lie. On-disk controllers lie. That's because these lies are caches, and cache buffers are one of the easiest ways to boost performance.

Apps are in fact crashing left and right.

My Materialistic app crashed about 30 seconds prior to opening this thread. I probably average about 5 "noisy" app crashes a week on my phone. No idea how many may be failing silently.

I had to give up on Materialistic because of the frequency of crashes. It's a shame because the hn website is really difficult to navigate on mobile (collapsing threads with that tiny + is a nightmare). Do you know of any better alternatives that let you log in and post from the app?

I have no issues with this guy’s client and have been using the free version for 3+ years. https://github.com/premii

Hmm, I checked that one out and it is indeed pretty nice but I don't think I can log in to my account with it. Just lets me view the public interfaces.

I login on the iOS app every day. There's a login option in the settings menu near the bottom.

Ah, I was using the webapp which doesn't have that functionality. Hmm, it appears the Android app is no longer in the play store. There's a dead link and a GH issue with no comments for two years.

I'll try to build the APK myself when I get some spare time.

Curious what kind of phone do you have? Its not very common for Materialistic to crash on me. It does happen but not that often.

Materialistic doesn't crash very often. When I say I have an app crash 5 times a week, I mean any one of my installed apps. If I ever come across a particular app that is crashing regularly, I'll look for an alternative and/or report the bug if the developer makes it easy to do so.

I have a one plus 6 and materialistic crashes almost every time I open it from a previous session. It also often crashes when reading articles even on a fresh instance

a couple of years prior to that era i worked for a Giant Electronics manufacturer who used WD desktop class drives in RAID and CCTV/DVR systems. we had insane corruption issues, burnout issues and so on with a large number of our devices. at the time i was running an rma shift for the product line so this was notable to me. we also had devices come in with Seagate drives, but they had been in operation much longer before they showed any issue and were from a few previous model releases. at some point the drives were "value engineered" and WD was selected. in the end they had to suck it up and put in a server class drive (something with a high duty cycle rating). once they did the fails slowed dramatically. when we went over the data it looked like all most of the failing devices were installed in 24/7 video monitoring applications, and several were installed in poorly (or not at all) ventilated spaces. after a while we put temperature tags on the inside of the chassis to get a sense of the max environmental temp (yeah, didn't work if it had been on a truck for week in transit, but at least a field tech could check).

if i had to guess, i'd say that the digital signage company was "value engineering" the system components ;) i was in industrial controls and never had a sense that the signage folks were laughing on the way to the bank...

It's somewhat implied it was one specific WD consumer drive model.

I used WD RE3s back then, and never had problems.

I had consistent problems with WD in that same era. Switched to Seagate as well, and my problems disappeared. Just another datapoint.

GP mentioned digital signage, so probably some displays at a business. Unlike a computer which would mostly either be left on (asleep) or shut down properly, I'm sure users had no qualms about killing power to these displays as they closed up every night. Write flushing problems would be exacerbated in that environment.

I doubt most consumer applications write to the database nearly as often as we were. Also very likely a particular model of the drive and also sheer luck that one worked and one did not.

See also: counterfeit SD cards.

Yes, I think it's possible (likely, even) that there was at least one more factor.

> because the company was pretty cheap, but also we had so many to deploy

Isn't "deploy many systems in hard to reach places" the prime target for enterprise-grade hardware? Fixing problems on one bespoke system is a lot easier than fixing the same problem 8000 times, and of course with a large sample size the number of problems experienced goes up.

I always thought "people willing to pay five to ten times market rate per gigabyte" was the target of enterprise-grade storage...

Yep. This was a big problem there. Before they were delivering content to PC's they distributed content via DVD once per month. DVD player failure rate was pretty massive, especially since they just started recycling refurbed players.

Contrary to popular belief, even consumer drives will honor flush commands. The problem is operating systems generally do not send flush commands, even when you might expect it (e.g. last I checked fsync() on Linux does not send a flush).

Disabling write caching resolves these problems. Unfortunately SATA drives typically ship with write caching enabled by default.

Of course fsync() sends a flush to the disk (unless you use non-standard mount options or "eatmydata"). They may not honor the FUA bit, though, but hopefully nobody relies on this if they expect it to work with consumer disks (See e.g. here https://devblogs.microsoft.com/oldnewthing/20170510-00/?p=95... ).

What I have heard is that some disks have problems if they are on a error recovery path. E.g. they have a corrupt sector or a sector that needs reallocation (pending sector), and then they have a power failure while the sector is reallocated.

Some cheaper HDDs and third-rate drives won't honor flushes. Same for SSDs, a few very cheap ones ignore flushes in favor of cache performance.

The implementation of fsync is FS dependent. On the default ext4 filesystem with the default options (barrier) fsync does send flush.

References?

Not quite the same thing but it turns out there are some other gotchas with fsync on Linux https://lwn.net/Articles/752063/

In my other life (2000-2008 ish I think) I've also dealt with the large (tens of thousands) deployments all consumer grade as in your case. No SQLite though. Obviously we've had some hard drive failures. What is very interesting is that the results were exactly opposite. WD drives lived and Seagate drives failed. I am wondering if such discrepancy was caused by the usage patter.

The drives weren't actually failing, the setup failed due to how the WD drives work. If you reformatted them they'd be fine.

Yep, exactly this. The drives were fine, only the database was getting corrupted.

Oops, sorry was not careful reading your message

WD: this comment Seagate: ST3000DM001

Another reason to stick to Toshiba or ex-HGST(if and when available)

Reply on news.ycombinator.com