Capacitor plague | Hacker News Reader

Capacitor plague

en.wikipedia.org

138 pointsllambda62 comments

This singlehandedly resulted in a company I was associated with not going public [March, 2010], when we had to take millions of dollars in write downs associated with our devices (which had some of these flakey capacitors) failing in greater than expected rates. I don't think it's an exaggeration to suggest that this problem materially changed my life, and altered the careers of many of those who were in our Manufacturing and QA departments (for the worse)

I also spent the better part of 2003-2005 replacing Dell Optiplex GX270s that had failed motherboards - bulging capacitors - almost 50% of our desktops were turned over.

Plague barely captures how bad it was.

[Edit: Apparently "Plague" was the word being used back in 2005 as well: http://news.cnet.com/PCs-plagued-by-bad-capacitors/2100-1041...]

I had to replace 280 small servers all over the eastern US thanks to this problem. We were just starting out so it came down to just two of us spending lots of time on the road.

I remember on reddit somebody talked about making a serious fortune by specializing in replacing those capacitors on numerous types of devices.

I made about $6000 in 2011 resoldering the capacitors in iMac G5s and selling them. It was only feasible because I had about 40 broken iMacs from my old job. But that was enough to pay rent for 6 months in Idaho. For everyone with dreams of VC capital and incubators, I highly recommend just scraping out a living any way you can if it's a stepping stone to your dreams. You can make as much in an hour as other people make all day, and use the rest of the time to code. It worked great until I ran out of inventory, now I mostly work from elance, odesk and freelancer. Taught me some valuable lessons though.

When I read "Dell Optiplex GX270," my memory tickled and I looked it up. Just as I thought. I used to be a field tech for a service company around 2007 and 2008. I remember having to swap the motherboard for a bunch of those computers in grocery stores. Same problem every time; popped capacitors.

Grocery stores run by Nash Finch, on the other hand (Sunmart, if you're in Nebraska) had old NCR POS terminals from the early 1980s. These were some of the very first computerized point of sale terminals in non-upscale retail stores. Those things were still going strong more than thirty years later. The printers on those things were mechanical failures waiting to happen, but the POS terminal, itself was crazy reliable.

The Midwest gets some impressive electrical storms. Occasionally, one of these storms would fry a peripheral card in one of the terminals. When that would happen, I would have to go to the store after closing hours, call up the tech who specialized in these terminals, and he would walk me through ripping out the proper card and rebooting that terminal from the master POS terminal. Yup, the thing would still boot up; even after a peripheral card had been burned.

NCR had long since gotten sick of that model and I don't blame them. Those printers were dodgy. Nash Finch were such cheapskates.

I had a Dell computer back in ~2005 that broke down literally 2 days before the warranty was up. I was dumbfounded as to why. Nothing had changed. I shut down the computer properly every time and it always started back up.

Two days before the warranty wears out, the "motherboard fries" according to the tech who showed up. I asked him him, he said it was probably static electricity. I figured it was some corporate conspiracy to bust computers just after the warranty was up and mine came a bit too early. (I mean... imagine if it had busted a day after the warranty! I'd be flippant!)

But this "capacitor plague" business makes a load more sense than that.

This also contributed to the failure of a startup I was working for in early 2000s. To be fair, it wasn't the only cause but it contributed to a loss of confidence from investors and partners.

On the plus side, you could get some really cheap electronics if you had a soldering iron and some replacement caps.

OT: is this a recently added Wikipedia feature?:

> On 18 May 2013, Capacitor plague was linked from Hacker News, a high-traffic website.[0]

Does anyone know the rationale behind adding this? It doesn't seem like they're auto-locking/semilocking the article to prevent inappropriate edits when an article is linked to from a high-traffic site.

0: http://i.imgur.com/LWwRQXF.png

It's not a feature just a template someone added. It's meant for talk pages anyway so I've removed it.

Can't say I've seen it before and I just read a Wikipedia article linked from HN last night[1]. I'm guessing there's a threshold of visits before their software flips that switch.

My interpretation is that it's a nice way of staying "Don't argue about the content of this page here, do it on your own message board".

[1]https://news.ycombinator.com/item?id=5722458

Edit: just saw johnduhart's reply.

I've seen a similar banner for years when /. links to wiki pages. It is just a template that someone pastes on to warn editors that traffic (and thus edit velocity) will be up for a day or two.

Ah, I see. I thought it was added automatically when a lot of traffic from one referrer was detected.

I always use this incident as an example whenever corporate espionage comes up. When you talk about security procedures, there can be a lot of eye rolling about how serious it is, but it can be pretty damn serious.

So you tell them to always make sure to steal the whole formula?

And do your best to test the damn thing. Stealing it can be almost as hard as inventing it (if turns out to be harder, fire your industrial spies).

Most motherboards and PSUs from this period had these faulty capacitors. I had a computer going bad because of this twice, once in the motherboard (Asus), then the PSU (Thermaltake). Luckily I learned about this and replaced the caps my self, but fixing electronics isn't the norm.

Think about all the electronic junk piled up because of this espionage slip. All the hidden costs and environmental impact caused. I'm sure even software faults could be attributed to that (at least the famous BSODs giving Windows a bad rep, I'm sure). Something impossible to calculate.

Perhaps the electronics industry would be in worse shape if our devices did not have these built-in self-destruction mechanisms.

The Viewsonic LCD 1600x1200 panel (2005) that I am reading this article on right now was a victim of the capacitor plague. Acquired it from a colleague who was going to put it on the curb. $5 and an hour of my time replacing 3 caps in the switching power supply (and an SMT fuse on the inverter board) restored it. I did it to add to my skills.

Paper dielectric capacitors were a scourge of older electronic devices, with service lifetimes even shorter than electrolytics. It is amazing how easily many ancient electronic devices may be restored to better-than-new operation just by replacing electrolytic and paper capacitors.

http://en.wikipedia.org/wiki/Film_capacitor

I had a pair of viewsonic VX922 that both failed with in 6 months of each other. Quick youtube how to and a visit to maplin (uk electronics store) got them back on their feet.

This singlehandedly almost and should have killed my startup. And the problem was so evil.. not all mainboards got it at the same time.. it was a bell curve distribution, and it started with simple lockups that a reboot could fix (which required a customer service trip). Then the lockups would increase with frequency until it no longer turned on, at which point my company had to replace it with a new mainboard and sometimes chassis for free.

Worst 4 years of my life.

A similarly annoying thing are whiskers from early lead-free/RoHS solder :(

There is at least one U.S. military contractor whose main line of business is reballing RoHS-compliant SMT components with good ol'-fashioned leaded solder. Because the military depends on electronics that HAVE to work, for a long time, in hostile conditions, and could give two shits about the environment. The Army fires depleted uranium into the earth, you think they care about a bit of lead?

To be fair, they're probably not aiming at the Earth.

Rule 4: Be sure of your target and what's behind it.

Aside from shipboard CIWS, it's pretty much always hitting the earth.

On the other hand, it's not that nasty.

You're aiming at whatever is in front of your barrel. Never point a firearm at anything you're not willing to destroy.

Besides, in this specific case, the whole point of DU is massive penetration, so of course it's going to eventually end up in the ground.

Just putting the uranium back where it came from

Actually oddly enough they're trying to not shoot as much lead into the Earth as they used to.

[1]http://abcnews.go.com/Technology/story?id=98594&page=1#....

[2]http://www.firearmsid.com/Feature%20Articles/GreenBullets/Gr...

[3]http://www.huffingtonpost.com/2011/05/04/green-ammo-lead-fre...

Assuming "green" ammo actually makes it into the field, we probably won't see it for a while.

I'm aware of one major consumer electronics device which was redesigned at the last minute for RoHS solder, leading to very high failure rates.

When you have a NASA failure lab implicating tin whiskers in the Toyota accelerator deaths we can call it more than 'annoying'.

source?

Was there any legal action in relation to this? If I bought capacitors from a company and used them in manufacturing my products only to find I have to replace large amounts of said product because I was supplied with faulty capacitors then I'd be looking for some compensation from the company I got the capacitors from.

I've recently taken up a hobby of fixing dead LCD and plasma TVs. So far, I've fixed every TV/LCD I tried, and so far, 100% of the issue was a blown electrolytic capacitor. It's kind of sad that a 50 cent part kills a $1500 TV. It doesn't take very much expertise. I didn't test the boards. I just looked for bulging caps and replaced them.

This is crazy, are there any comprehensive lists of devices that were affected by this? I am a little curious to see what devices, of many that have failed, may have been affected by this. Not that it would really change anything at this point.

I probably used to have a stack of devices just in my house that would qualify for these, from desktop mobo's, graphic cards and routers. I've probably lost $5k personally over the last 10 years to this, of course almost always out of warranty period.

It wasn't just computing equipment. I have hand-repaired a dozen sets of amplifiers and powered studio monitor speakers made between '02 and '06 that just needed $2 worth of caps replaced despite the manufacturer quoting $250+ for authorized repairs.

In about half the cases it was a combined failure of the caps being el-cheapo and also the board design placing them far too close to heat-generating components. Fortunately, it is still cents on the dollar to replace with a higher temp-specced part if you know how to wield an iron.

Basically it's a good deal of the devices in the "iMac G5, Athlon XP, Pentium-4 with HyperThreading" era, or at least that's when I saw it the worst.

Laptops generally weren't affected, as SMD capacitors generally aren't electrolytic.

I used to do desktop support for a University, in which my daily job was reclaiming and sifting through hordes of gx270's and gx260's to find working mobos.

I swear they made the cases on those things out of razor blades. I probably could have replace the electrolytic material in most of the blown caps with my own blood.

I have lost some Pentium 4 motherboards (and video cards from the same era) to this. Don't remember the specific models, sadly.

Not comprehensive, but the premade recap kits at badcaps.net has a lot of the more popular devices listed.

Additionally, I've seen at least three home routers from friends and family fail because of bad capacitors--visibly blown, and in two cases I replaced the caps and they started working again. (In the other case I just threw it out.)

http://www.badcaps.net/pages.php?vid=21

I got pretty good a soldering around this time. Scored a free "broken" monitor and spent about 75 cents for new capacitors :)

How do you test for a broken cap? Is there any tutorial on desoldering and soldering them?

The easiest test is visual: if the surface is bulging even just a little and/or the capacitor isn't standing straight (this one doesn't apply to Soviet electronics), it has failed. There are lots of tutorials on the net, there is definitely something on http://www.eevblog.com/ But the way I learned it was to just take an old PSU, try to get a component out and then solder it back on.

[edit:] Some googling found this thread for capacitor testing: http://www.badcaps.net/forum/showthread.php?t=17592 I found the badcaps community pretty good when looking for a new PSU, there seemed to be many knowledgeable posters.

I was once able to fix a failing motherboard by cutting out the "bulging top" capacitors and soldering new ones of the same spec on the parts of their legs that remained on the board.

That was after almost ruining it by attempting to actually unsolder the original caps from the board. Those things are soldered hard! I tried to apply too much force, slipped, and made a big scratch on the board with the soldering iron. Didn't go through the varnish though.

In 2006(ish?), I worked for a subcontractor doing hardware swaps on Dell machines, and these capacitors single handedly accounted for maybe half of all repairs that we had to do during the time (and that's against even trivial and common stuff like laptop keyboards breaking). It really doesn't surprise me that it cost Dell $400M+ to fix these systems - it had the sort of impact that could definitely destroy smaller companies outright.

This seems to refer to a specific "bug", a bad electrolyte, but doesn't this eventually happen to all electrolytic capacitors?

The "bug" is the accelerated rate at which it failed, not that it failed.

Many of us have electronic devices from the 1980's and early 90's that haven't failed.

Until Congress sold the television spectrum and the FCC ordered broadcasting to cease, many of us still had non-HD TVs.

Until Congress sold the television spectrum and the FCC ordered broadcasting to cease, many of us still had non-HD TVs.

Um, I still have mine.

I only use it so that the kids can watch DVDs. There is still a good enough selection of DVDs available. I'll upgrade when it breaks and is irreparable. But not until.

Even well made electrolytic caps are often the shortest lifespan components in a design, so yes, they are still a problem. But this was also a poorly chosen electrolyte.

It would be nice if the article included a comparison of the expected and actual lifespan of the capacitors...oh wait it does:

"Many of the capacitors had a life span specification (load life) of 2000 hours at 105°C. With a lower average internal temperature of 45°C on a printed circuit board and a ripple current within the data sheet specifications, these capacitors should have a life expectancy of about 18 years of continuous operation. With respect to this life span expectation, a failure after 1.5 to 2 years is very premature."

Yes.

If this happens to your Samsung LCD TV, it is a pretty fun project to fix yourself! the only thing you need to is to order to the parts, youtube, and a soldering iron.

I personally had two motherboards with this problems and one monitor. I also had 3 graphic cards. Only two of the graphic cards were still under warranty so I had to eat the cost of most of these premature failures. We also had numerous Dell computers at work fail but they were all out of warranty by the time they failed. Knock on wood but my last item failed in 2011 (graphic card) so hopefully this is over.

I completely forgot about this, but in high school I did IT for my small school district, and I definitely remember having to replace a huge number of motherboards due to failed capacitors around that time. I think they were all HP computers, and at the time I just assumed it was one bad run from HP's factories, but it must have been this.

I'm really unconvinced that this continues to be a current problem, and the Wikipedia article is definitely not clear on this. If there are recent examples of this that are clearly documented, it would be great to see some good hard data or sourcing, either here or perhaps the Wikipedia article's talk page.

In the introductory paragraph, at the very top of the Wikipedia article, citation included:

"As of 2013 the problem seems to have receded, with the last major surge of complaints being reported in 2010.[7]"'

[7] http://www.ronhoutman.com/the-capacitor-plague-strikes-again...

The estimate of 2007 being the end of it assumes the devices were plugged in and being used the whole time, right? I still see bad caps happen at some of our clients, but they are old machines that aren't powered on all the time...

Yep, had an iMac G5 with this problem. Apple shipped me an entire replacement system board assembly. Later they started having people take them into Apple Stores for the repair.

http://www.flickr.com/photos/mrbill/62059256

I still have a Compac 450 workstation that probably would blow up if I turned it on. Got it from BBN when I worked there. Many of the others back then died this death. The thing is a tank! Which is mostly the reason I still have it.

I've had several devices blow up or catch fire because of bad capacitors in recent years. These were not very old devices. I'm surprised there haven't been more serious consequences than what has been reported.

Is this the same problem that affects first-gen Xboxes?

http://www.jaguarsector.com/index.php?showtopic=33652&hl...

i had an original apple airport (UFO style) that blew a cap. this happened my freshman year of college, which was 01/02. wow. memories.

https://www.google.com/search?q=apple+airport+ufo&safe=o...

to apple's credit, they replaced it without cost. seems like they sent me a refurbed unit with "good" caps.

Reply on news.ycombinator.com