Hacker News Reader

Comment by epberry | Hacker News Reader

I used to be surprised when people came across these bugs. I would think: a kernel bug! They must have truly hit an edge case for something like that to skip into production. But then I read Showstopper: The Race to Build Windows NT, specifically the part where that OS shipped with 65,000 known defect (and it was considered one of the most stable software releases of all time), and it became clear to me that these bugs are probably known by the team when the OS ships but just not considered high priority enough to fix. Same with the recent articles about iOS and engineers not getting time to fix P2 bugs. Long story short my entire perspective on kernel bugs (and partly software development) changed and I began to think of them more as known flaws that weren’t fixed because of schedules rather than unknown bugs that were uncovered in the wild.

View on HN

that OS shipped with 65,000 known defect

Issue tracking at Microsoft (and presumably all large tech companies) includes things like:

* localization errors

* typos in user-visible text

* suggestions for rewording user-visible text

* issues related to icons and other graphics

* issues related to sounds that ship with the product

* feature requests

* feature enhancements

* tracking tickets for work to be done or work in progress

* tickets for unverified, incorrect, and unreproducable bugs

So a. 65,000 tickets, not 65,000 defects. b. "65,000 tickets" is meaningless without knowing more about them.

I found my own one: Windows "structured exception handling" relies on code in the kernel that unwinds call stacks. This makes a whole bunch of assumptions about the layout of the code as well. On CE/ARM, we found that Visual Studio was sometimes generating code that used "bl [r3]" constructs which could not be unwound. The workaround was to use a different version of cl.exe that came with the CE dev kit and not the one that came with Visual Studio.

Oh, and if you have more than about 512 bytes of local variables in a stack frame and try to use CE remote debugging, that stops working as well.

Its the same on X64, although I think its better documented/understood on that platform.

Yeah that's one of the things Cutler played an important role in -- formalizing x64 prologue and epilogue constructs. The epilogue stuff is interesting, there's a small set of instructions that are allowed because the unwind code literally does opcode analysis to figure out if the current function is in its epilogue or not.

Heh - that was partly me. The kernel team (i.e. mostly DaveC) mostly prescribed which instructions were allowed in prologues/epilogues. I was dev lead of VC's CRT team at the time (2000-ish), and got assigned to work on that and come up with an x64 version of the prologue/epilogue descriptors needed to unwind. I looked at what was done for Itanium, and simplified that waaaay down to what was needed for x64. I remember designing the ability to split prologues into multiple parts, to do something called shrink-wrapping, where you might not want to push all your saved registers all the time - there might be a subset of a function's code where a register needs to be pushed/popped. DaveC needed convincing that was a good idea.

This is one aspect of OS development in which I think open source wins: users are not at the mercy of the OS developers for bug tracking and/or fixes.

Are we not? If I find a bug, I sure hope they fix it soon. I do not have the time and energy to work myself into the kernel to fix the bug (unless it is a small bug that does not require a lot of knowledge about the kernel itself).

Sure, as an individual, your direct powers are limited, but they're a lot less limited than if your kernel is closed. You can offer a cash bounty to anyone who wants to fix the bug (not just those employed by the kernel-vendor) and you can deploy their fix to your own systems independently of the kernel vendor releasing an update (or, accepting the fix at all) because you can build the kernel from source yourself (or, again, you can pay someone to do this for you). If you can't afford a big enough bounty to get anyone to bite, you can team up with others affected by the bug to pool your resources, and you can even use the kernel bug tracker to coordinate this effort.

Good points. Thanks!

At the very least you can write a bug report. I tried writing one for Microsoft Office, not even found where.

I think they consider that a feature, with their 1 billion (not-really technical) users :)

First off, true, reporting bugs to Microsoft is hell.

But on the other hand reporting feature suggestions to Microsoft is much more pleasant than anything open source has e.g. https://office.uservoice.com/forums/285186-general

In Open Source world, because by the nature of it you're non-paying, asking for features either gets back "submit your pull request" or "go away." Which makes sense, those who contribute the most time get the most say, and that would be fine if those that contribute the most time had the same goals and taste as those that didn't (see the failure of Linux Desktop as an example).

Looks like people really want ClipArts back.

It was nice having a license free library of art assets even if they were overused. There are websites that offer that these days but as some of the commentators pointed out, that requires legal to sign off on in some businesses or those sites could be hosting stolen images.

> It was nice having a license free library of art assets even if they were overused.

I guess by now Emoji fill that role to some extent. Including the overused part ;-)

How many times has this actually happened?

Here's a bunch of >$1000 bounties posted: https://www.bountysource.com/bounties/search?direction=desc&...

None for the Linux kernel, though, but it does have >20,000 forks on GitHub: https://github.com/torvalds/linux/network/members

Yeah, but if you're a professional it's just part of your job to patch the kernel when you have to. In any case, the kernel isn't as scary as most people think. Nor is almost any other "arcane" software in your stack.

I think in reality it's more nuanced than that: the number of people who use computers who can fix a kernel bug is relatively small, and the process of getting a bug accepted, patched, and shipped is not trivial. I would not want to bet on a bug which Linus disagrees with getting patched faster than, say, a report on some company's enterprise support agreement.

(disclaimer: I've been running Linux and other open-source OSes for over two decades. I like OSS but it's not perfect and staffing for projects is an under-solved problems.)

There are many small bugs in the kernel that are not hard to fix but are not priorities of many kernel developers, but the maintainers are usually happy to take your fixes if you create them. Proof; I've submitted a number of small fixes over the years that have been accepted: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Are data corruption defects treated the same way as other things? i.e., would kernels get shipped with known defects that cause data corruption under load? I could see kernels being shipped with known cases of hanging, crashes, out of memory, etc., but the same for data corruption under load would be a little hard to swallow.

Bugs are prioritized. Data corruption would be higher than some other bugs, but it also depends on frequency

Yes, but do you know of any large software project that only ships when they hit zero bugs? Actually, this would probably extend to all products. Creating a product with no known defects is not possible AFAICT.

If 250kLOC is “large”, then the Space Shuttle’s onboard software is one (the only?) example.

https://www.fastcompany.com/28121/they-write-right-stuff

This is the case with all major software releases. The bug tracker is filled with bugs for any major release. It's just a matter of team lead, project manager and higher ups deciding what the priorities are. Not every feature makes it and not every bug gets fixed.

The key is to hit the milestones and get the product working as much as possible.

Enterprise Software Motto: "Sometimes good enough is good enough". Amen.

Think about your code. Now imagine a codebase 1000 times as large written by thousands of engineers with deadlines. There are bugs or potential vulnerabilities in every codebase, but you’ll never encounter the vast majority of them.

Reply on news.ycombinator.com