Hacker News Reader

Comment by nabla9 | Hacker News Reader

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS.

When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening.

In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once a month or so. Oracle had to send on-site team to monitor the database operations to catch the bug in action. They basically had their own datacenter to record all the logs from database, networks and operating system.

It took weeks for them to catch the bug. It was related to certain network driver in the operating system and the bug manifested only with some specific network traffic pattern.

View on HN

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip.

But the issue was, this was happening even after we removed all instructions to delete data from the flash.

The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness when the flash was connected to controller outside the board.

The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died on average. But we had millions of them in the wild and we would get hundreds or thousands die on our customers every week.

It took half a year of debugging and tens of millions of dollars in losses.

We found that one of unrelated components on the board emitted too much noises. And a software change to speed up writes to the flash removed a special sequence of that preceded and followed every one word write to the flash. This special sequence protected the flash from accepting a random noise as a command. Usually, the command would be something harmful or meaningless. But sometimes it would be interpreted as a real command -- and one of these was a command to clear entire flash chip.

I forgot to mention the best part of it. The fix was a one character code change (to enable a feature that we have disabled before).

Another source of bogus commands can be brownouts. When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring.

>> When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring.

I heard from a coworker that an older version of the product had bumped from a 16MHz chip to 20MHz to run more code. Chip supplier screened parts for the higher speed, but didn't cover higher speed across the normal voltage range. When supply dipped (within the old spec) switching speeds slowed. The first circuit to fail was something in the branch condition logic. This caused all conditionals to not branch, which lead to running all code in the memory space in order. That included a few instructions to.... fire the airbag. It was a rare thing, but a very big deal.

I wonder if anyone else instantly gets a visual after reading your story:

“Business woman on plane: Are there a lot of these kinds of accidents?

Narrator: You wouldn't believe.

Business woman on plane: Which car company do you work for?

Narrator: A major one.”

I heard the story decades ago, and the problem had been fixed some years prior to that. There are undoubtedly hundreds of similar issues across the tech industry, most get quietly fixed before much damage is done. I only tell the story because it's an example of how obscure details come together in unexpected ways to cause failures. That happens everywhere.

And how brittle the borders we programmers see a strict, as a branching condition, "this should never happen". But in hardware, things are different.

How do you even protect against such a thing you just described? Put the airbag on a diffent microprocessor?

Maybe put some code directly before bag deployment code that disarms the system. So even if the execution reaches it directly through the branches, nothing will happen

Sure but what if the address bus glitches past that block?

Maybe put arming code somewhere completely different?

Of course it's possible to create more and more improbable scenarios. That's not to say those never happen; I've seen really improbable occurrences.

All the stuff that counted on our board had brownout protection. And if it didn't it is relatively easy to deal with (as long as you are aware of the problem).

(We may have learned a bit about noise and improved further iterations of the device, but sw change was enough.)

I think something similar happened to my sd card at a hackathon years ago.

We couldn’t change the data on it. Linux would report that data was written to it, but when we unmounted and remounted it, the data was unchanged.

https://devpost.com/software/xcaliber

Oh god, that's insane

epic debugging story

Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.

1. What does choice of MCU have to do with it?

2. Noise can definitely affect other components. Traces are antennas that both broadcast as well as receive signals.

As technology marches forward, the voltages and tolerances for communication become smaller and tighter. At the same time, you integrate more functionality on the board so maybe your sensitive MCU and flash and ram are all living close by to some radio equipment and PSU and whatever else. Power interruptions to certain components can create momentary flashes of noise that can act as EMP bomb going off. On other occasions a noisy, badly designed high frequency power component can emit noise that is picked up by some random traces due to their geometric configuration and length.

Your comment suggests you never worked on a large project with thousands of components and dozens of major chips?

from the description the flash was on a separate chip: I would guess most likely a SPI interface. You can absolutely get noise picked up on a SPI interface which looks like a command: and if an erase command is 16 bits long, you are going to see that command just with random bits every so often.

Yes, it was separate chip (per description). Yes, it was SPI flash. And yes, the word was 16 bits.

>The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died

That's one failure per 16,800 hours, or 700 days. That's pretty goddamn rare.

Once you have 700 units in the field, one failure per 700 days is no longer quite so rare.

And that’s running the particular op repeatedly, so in practice it’s an even rarer event. However with “millions of them in the wild” it doesn’t matter. Scaling is tough.

We had to clean out a small warehouse and repurpose it for testing rig. We have created a piece of hardware to do automated testing and recording, and then we created a small production chain to and produced 100 of them.

Fortunately, we already had all the design, development and production resources to do the project on site (yes, we were doing it all, not outsourcing to China).

First, you may enjoy [1].

Second, a story from many years ago.

The local ACM chapter I was a member of bought a whole bunch of e1000 NICs, back in the day when 1Gb at relatively affordable prices was new, and a switch, and threw them in our servers and desktops, which were a motley assortment of Intel, AMD, and Other things, running various flavors of Linux and Solaris.

_Some_ of the Linux systems had a problem where eventually, the NIC would stop responding and dmesg would flood with "Tx Unit Hang" or "Rx Unit Hang", and not work again until a reboot.

If we swapped the NIC from a system where this never happened to one with the same model NIC where this happened, the problem kept happening - that is, it was something about the system+NIC, not just the NIC.

Intel eventually gave us a shipping label to borrow one of our systems when they said they hadn't been able to reproduce it but kept having people report it, and we said "we have a number of them".

They made some really exciting noises after some debugging, and eventually concluded the issue was that our really cheapass Sempron board that we sent them screwed up badly if you tried negotiating 64-bit PCI in their normal PCI slots, and would sometimes miss the messages between the NIC and the host saying that Tx Ring buffer 0 is free/full, etc, and eventually, it would miss so many of those that it would think the NIC had no more buffers, and here we are. They couldn't reproduce it because all their testbeds were well-designed and tested things.

Choice quotes mid-debugging include "I do not see how anyone could make a PCI device work reliably in this system".

They did figure out a workaround, and ship it, and asked if they could keep the system permanently for future testing.

But that was exciting.

[1] - https://github.com/openzfs/zfs/pull/15588

A second fun one.

In a previous life, I was paid to build storage systems for some HPC-ish workloads.

So I was testing a bunch of cheap-ish desktop drives that had really nice (for the time) sequential throughput, supposedly, and threw together a Supermicro system with a Xeon, ECC RAM and some SAS HBAs and several external enclosures full of these disks, made a couple of raidz3s, and started trying to stress it.

I quickly found that rarely but somewhat reliably, I'd get correctable checksum errors, and because it was raidz3, it always had a lot of spare recovery bits, but the numbers kept going up...and went up even after a scrub finished and "corrected" all of them.

Well, SAS has checksums over the wire, so I'd be seeing disk errors if the wires were eating my bits, and it was across all the disk controllers, so either they were all bad or it wasn't the controllers, ECC RAM and no correctable or uncorrectable events fired...

This predated ZoL, so this was originally on early illumos - I then tried FreeBSD, and it did the same thing.

Huh.

So, the drives in question were Samsung HD204UIs, which, it turns out, have a really spicy firmware bug, where if you send them a SMART IDENTIFY request with data in the write cache (e.g. it already told the OS it was stably written out), it just...dropped it.

So the background smartd I had running for collecting data on the disks was causing them to eat some of the writes whenever that lined up.

The funniest part was, though, that Samsung released a firmware update, but it doesn't change the reported firmware revision, so you can only know by testing if the disk is going to eat your data like that.

I believe smartctl still prints a big warning to this day about all this if you ask it about those drive models.

One more for the road.

So, I have a little old SPARC64 niagara 2 box, which is a strange beast for a number of reasons.

The onboard NIC is a strange bespoke Sun-created Ethernet card, over the PCIe bus.

Since it's a SPARC in 202x, it's not really well tested or supported, so updating is always fun.

One day, I lost power, and on next boot, it booted a newer kernel, and the kernel panicked.

I rapidly determined this was from trying and failing to initialize the NIC, and attempting to bisect it turned out to be really exciting - after getting impossible bisect results, I realized, once I booted a "working" kernel, all successive kernels worked until a cold boot.

The problem, you see, was that they had changed some things about the PCIe initialization to enable certain kinds of memory protection features.

The actual memory protection feature works fine on this hardware - but the _check_ of whether the feature is supported causes the machine to panic.

But once the machine is booted, it doesn't try to re-initialize this on warm boot, so it doesn't trip this problem.

Disk drives really are the worst. Someone who works at Google or Amazon or The Internet Archive could (and should!) write a book where every single chapter tells the story of discovering some new batch of hard drives with a new and crazy firmware bug.

> certain network driver in the operating system and the bug manifested only with some specific network traffic pattern.

Found a similar issue in a switch stack relating to multicast stream delivery. Knowing about this class of bug and having isolated it down to the switch, as the only possible device in the path, being responsible, I filed a ticket with the appropriate team to get the stack rebooted.

They were understandably skeptical of my supposed prescience and so I had to spend a considerably larger amount of time writing test cases and running deep packet captures in order to prove that a single bit, in the port field, was getting flipped under circumstances that were repeatable but not entirely explainable without even more work.

Finally, they relented and agreed that, just turning it off, then on again, was indeed the best initial solution. Fortunately with all the test harnesses in place we were able to prove that it cleared the issue on that set of ports.

Network drivers contribute to "bugs" surprisingly often. I've had my share of issues with them.

I'd go as far to claim this: If you're shipping a product that relies on networking at your customer's premises you basically need to maintain a list of cards and drivers that you support. Especially if you're doing anything weird, where "weird" means anything but bog standard TCP.

How can network driver corrupt a database?

Most network drivers run in kernel context and can access and write to everything.

Additionally, unless your system has an iommu and configures it to strictly limit your NIC, most NICs can DMA to any address in memory.

Unintentionally providing the wrong address to receive a packet with is a great way to corrupt a database.

By corrupting the data that was received.

I never really accept bug resolutions like that. Oracle corrupted data because a network driver fed it bad information? Did this network driver manage to preserve the legality of the corrupted info? Unlikely, so why did oracle believe it? If the corrupted message was still legal, then it sounds like the protocol isn't robust.

I'm sure there's all kinds of nuance to your tale, but it still sounds like Oracle's bug to me. They just found what it was in your system that introduced the behaviour they hadn't anticipated, and then got you to remove it.

network drivers can usually corrupt arbitrary kernel memory

Reply on news.ycombinator.com