Why Windows 95 and 98 would crash after 49.7 days of uptime | Hacker News Reader

Why Windows 95 and 98 would crash after 49.7 days of uptime

sites.google.com

76 points0xedb81 comments

This is a pretty uninformative article. It doesn't actually say why Windows would crash, as the title promises. It does some trivial math, points out there's an overflow, and then says "Clearly some code in Windows didn’t handle this too well, and Windows hung."

I don't agree. Nobody of the readers would care about the programming error made 25 years ago in some closed source code.

However, timers wrapping around is a very current problem every programmer should be aware of. Not too long ago we had a problem with corrupted video files. After weeks of investigation it turned out that some code used an 32 bit unsigned to count microseconds. Well, that wraps around after approx. 1 h 10 min... Obviously most test cases were videos shorter than that.

I'm a reader and care, so at least one.

Fair enough. If you stick to open source, then you can always know how programmers got it wrong in the past and present.

No, that allows you to know how programmers of FOSS got it wrong.

Not all programming is the same; there are many lessons to learn about software engineering (mostly in the "what not to do" sense) which, for practical reasons, have only ever showed up in (time- and budget-constrained) closed-source codebases.

Do you you have any fact-based evidence that certain bugs/anti-patterns would happen less in open source code bases or free software code bases?

Well, security by obscurity might happen a bit less. And hopefully many bugs in contributions are spotted instead of being merged because the author thinks it should work. But all major open source projects have had amazingly "stupid" programming bugs that went undiscovered for many years. Or the same feature has been patched again and again, but still causes more trouble.

Knowing that many company-internal software projects work with little review and very low testing coverage, it should be natural that their code quality is even worse. But that certain kinds of problems would only occur in closed source does not sound evident to me.

> Do you you have any fact-based evidence that certain bugs/anti-patterns would happen less in open source code bases or free software code bases?

How could I have that, or the opposite of that, without access to closed-source code-bases?

That was my whole point: that you can't know what sort of selection bias you're implicitly accepting by only looking at a certain easily-accessible slice of the data.

In order to know that it's safe to assume that FOSS is representative of all software, you first have to get access to enough non-FOSS codebases to be representative.

Then tile is misleading.

This is a running theme on HN:

Why X does Y?

Well we don't know but let me talk abut Z in relation to X.

Because they didn't handle an integer timer wrapping around correctly. That answers most of the question. What remains is did they forget altogether or program it incorrectly. I'd say I have seen worse click bait, this post didn't annoy me even if the analysis was much simpler than a Google project zero blog post.

Did you seriously expect to see Windows source code published? AFAIK it is still under strict NDA regardless of how old it is or did I miss some headlines?

I've read several blog posts that actually go and analyze why a bug happens, by disassembling DLLs or whatever they need to do. So yes, with a title like that, I do expect more than "oh, it's probably related to integer overflow somehow."

Not just windows 95 and 98 and it would have to be something buried in the OS for a 32 bit counter going to zero to freeze the system.

"Microsoft server crash nearly causes 800-plane pile-up"

https://www.techworld.com/news/tech-innovation/microsoft-ser...

Nowadays you don't even get CLOSE to 49 days uptime before Win10 forces you to restart due to updates ;s

Well, that's one way to the fix the bug ... ;)

My laptop with Win10 is at 72 days.

I created a bat file with:

  net stop wuauserv
  sc config wuauserv start= disabled

and then scheduled it to run every 10 minutes. The problem with that is you now have to remember to disable it and check for updates when you have time, which I'm obviously not doing.

You're oddly proud about not having updated your OS for 72 days.

No, I'm proud that I haven't rebooted in 72 days. The last clause was me acknowledging that this isn't good computer hygiene.

edit: And I'm trying to help other people. I know how infuriating it is to have your computer restart in the middle of something, and the above fix is pretty straightforward.

I think having to disable the autoupdate in win10 is the most requested thing for IT helpdesks these days.

As a result of this complete ignorance to ergonomics, the windows ecosystem is returning back to WinXP years, and windows users are now getting pwned by years old exploits again.

I think microsoft best achievement in win7 was that it was working "out of the box" without having to hack it. Now, win10 again requires to be hacked right out of the box just to work.

People now install stuff like https://www.novirusthanks.org/products/win-update-stop/ or https://securityxploded.com/windows-autoupdate-disable.php and countless others.

But when they finally need to update, the half broken windows install may give up the ghost, because the update script can't account for hundreds of ways how windows internals may have been altered, and so they learn to not to update at all, and switch to antiviruses, qihoo and other "crapware clearners" in the end

> Now, win10 again requires to be hacked right out of the box just to work.

Or, to rephrase that (from Microsoft's—and most IT departments'—perspective): Windows now works, as once again users with broken workflows are now forced into fiddly sub-optimal patterns for using the OS, which are more work to maintain than just correcting their workflow. (And IT departments are empowered to not even allow these fiddly sub-optimal patterns through GPOs, allowing them to force users to immediately rectify their workflows.)

Microsoft is trying to be the teacher that slaps your hand with a ruler when you look down at the keyboard instead of touch-typing. Except, instead of touch-typing, they're trying to force people to use (and build) software in a way that makes restarting their computer irrelevant, by e.g. making long-running jobs checkpoint their work and resume from the last checkpoint after a restart.

Still doesn't work for all cases. At my work we had a system running a control loop reboot due to updates, which reset a system which had been stabilising for weeks (we had tried to stop it updating, unsuccessfully as it turned out). Windows is basically rendering itself unsuitable for running such a system because of their update policy (which is a shame when the systems necessary for running the control loop depend on it).

I mean... shouldn't those systems be running Windows IoT Core (nee Windows Embedded)? That branch doesn't have auto updates, AFAIK.

These systems shouldn't be running Windows to begin with

These are lab experiments. The people running, configuring, and setting them up neither have the expertise nor the time to set up such a system.

I can't say this chimes with me at all. I have seen plenty of Windows server 2016 (W10 for servers ish) stay up for over a year. On an AD network you control the update policy! I can stop all of my clients updating with a group policy.

This is basically a desktop system running a lab experiment. After the pop saying 'reboot in x hours' we changed the group policy, however this does not stop a pending update.

> I think having to disable the autoupdate in win10 is the most requested thing for IT helpdesks these days.

That has not happened even once to me, and what company allows users to decide?

I can tell you that the majority of calls to my desk from mac users are a direct result of not updating. The Windows machines all have their updates controlled by WSUS, a standard component on an AD network. I have very close to zero problems with Windows 10 updates.

> how infuriating it is to have your computer restart in the middle of something

That last happened to me a couple of years ago and I used Windows 10 every day since.

Or that you have to jump through hoops to get your OS to do what you want it to do.

About 15 years ago, I worked at a startup that had a server with 5 years of uptime!

I have a home server with 1301 days of uptime.. (last year it survived a move without rebooting)

That's amazing! The most I've made it with a home server was about 500 days. Eventually, there would be a thunder storm and the UPS would die. Then I'd use the reboot to do all the updates I should've done in the past year and a half...

> (last year it survived a move without rebooting)

How? UPS?

Yep, moved the computer while together with its UPS.

that's very cool, but what the heck is on this server that made it worthwhile to move it like that?!

Probably his uptime record!

If it's a short enough move I can see doing that just for the hell of it.

IRC bouncer with unregistered nick.

DEC computers were know for their reliability.

This was a generic, rack mount Intel server running Debian 3.x.

I seen now that my comment was out of context. Sorry.

I was referring to longevity in general. Back in the day, DEC computers would just about never go down. They easily had up times of longer than 5 years.

No prob... That is true. I used to do some work on VAXes and they were very reliable.

Choices are:

Allow Windows 10 auto update but for a chance for it to delete all your profile / data files PERMANENTLY :

https://www.bleepingcomputer.com/news/microsoft/windows-10-k... (February 19, 2020)

Turn off auto update with user control on when is best time to update:

Make backups, would you?

Useful, I may purloin that.

I don't object at all to regular reboots for security updates, but I do object to forced reboots. Windows not being safe to leave alone for 24 hours without risk of it rebooting itself makes it no better than a toy OS in my mind. And it isn't as if it is just the one monthly reboot around patch Tuesday plus the occasional emergency patch: I think I can count on one hand the number of months in the last ~two years when my main Windows machine hasn't needed to reboot twice or more in a month.

Actually I do object to regular reboots if I'm more honest, at least as many as Windows seems to need. The fact that just about any security update requires a full system reboot instead of many of them being applicable with just restarting component services smacks of bad design.

In the late ‘90’s I used to brag about my Linux uptime being over a year. And that was a system was directly exposed to the internet. I can’t imagine how many unknown security holes it had during that time.

These days it's still possible to patch and have a high kernel uptime due to in-kernel Kernel Live Patching (KLP) support. I've written a few KLP patches to solve kernel bugs without rebooting just because rebooting every system takes a long time to complete.

I'm not a systems person and pretty new using Linux, so have a lot to learn. I have Ubuntu installed on a computer here at home and recently enabled Canonical Livepatch. I also update whatever needs updating when I SSH in based on the "x packages can be updated. x updates are security updates." messages I get. Some of these require a restart. This isn't a problem for me since I'm just messing around. I'm wondering how this is done in a production environment?

I assume that a lot of the packages I update either don't need to be updated or aren't getting updated on a production server, and therefore less restarts are needed?

Go to /etc/apt/apt.conf.d/ and have a look at 50unattended-upgrades and 10periodic. My 10periodic looks like this:

    APT::Periodic::Update-Package-Lists "1";
    APT::Periodic::Download-Upgradeable-Packages "1";
    APT::Periodic::AutocleanInterval "7";
    APT::Periodic::Unattended-Upgrade "1";

And 50unattended-upgrades:

    Unattended-Upgrade::DevRelease "false";
    Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
    Unattended-Upgrade::Remove-Unused-Dependencies "false";
    Unattended-Upgrade::Automatic-Reboot "true";
    Unattended-Upgrade::Automatic-Reboot-Time "02:00";

No more worrying about updates :)

The highest uptime I've ever seen was a 11yr and change on a Cisco switch on a state university campus. We made up some little commemorative plaques with a CNC router and affixed one to the case before eBaying it. Hopefully it delivered reliable service to the second owner.

This is a myth that cannot be proven or disproven. Nobody was ever able to keep Windows 95/98 up for that long.

Jokes aside, around 2001 I was asked to do some maintenance on a machine running Windows 3.11 that was running an app to control all access within the building. When I asked about potential upgrade the guys balked at me saying the machine has couple of years of uptime and they are not very keen to have to restart it on a schedule due to windows 95/98 bugs.

That's the reason we still have COBOL and similar systems working.

For exact this reason. They are working and doing what they meant to be doing.

This brings up a good meta point: people who like to complain about the quality of software in the modern world almost certainly didn't live through the early[1] days of the industry.

This sounds crazy to modern ears, but at the time the idea of a desktop system actually staying booted for a whole month was weird and alien. They just didn't do that. Commercial unixes were better. It was routine to find servers that had been running for three digit (!!) uptimes, and people would brag about stuff like this. Linux, when it arrived, was in this category too and we'd all puff to our friends about how we "never turn off" the machine under our desk, because it would always be able to run for a month or so before it crashed.

It was just a different world, and things, despite some of the community emotion, have gotten vastly better. We're much better at writing software than we used to be, which tends to make some of old folks a little confused when people hold up "new" ideas as being paradigm shifts in software quality.

We already had the shift! We're just picking the higher hanging fruit now.

[1] Do the mid-90's even count as early?

I have a Hackintosh running High Sierra, and at most I reboot once a month.

<rant>

The problem is not people writing buggy software or them crashing. At least for me problem is we keep on reinventing the wheel, without corresponding increase in productivity. I don't see people cranking s/w better or faster then they did with Delphi in 1998. Instead of standing on shoulders of the giants we disrespect them, our whole industry is built on abusing all the hardware and technology gains for frivolous wars and gains.

</rant>

Well, there are huge differences (in the MS world) between 9x and NT.

DOS (before) was very, very stable (but of course was not multitasking) and widely used in industrial setups.

Windows NT (before Win9x) and Windows 2000 (right after) were both, very, very stable, I had machines running 24/7/365 that were rebooted once a year or so and for other reasons (maintenance/update, hardware replacement, black outs longer than UPS capability, etc.).

DOS wasn't functional enough to be described as stable; it did almost nothing but look after the files on your file system and provide a few APIs, via interrupts, for programs to read and write files and perform console I/O. Stability was almost entirely the responsibility of the program you ran on top.

I do often complain about the quality of software but that is because i complain about the quality of most of the software. The foundation we work on is usually much more robust than what it was in the 90s, it is the stuff that we build on top that is shaky.

And a big issue (at least on the desktop) is that in many cases we keep throwing away old stuff and reinvent new stuff with their new bugs and issues, instead of fixing that old stuff (and i do not think that it is a coincidence that most of the stable foundation is actually old stuff that had a lot of people trying to fix its issues instead of throwing everything away).

Coding in DOS, saving before executing your program became a reflex because any hard freeze in your program would force you to reset the machine. It wasn't rare, and practically unavoidable if you're trying to do something clever like a TSR. Or doing something interesting with graphics - if you got left in a weird mode, you might have to type blind to recover, or give up and reset.

In the early days of web-hosting providers, it was common to see a version of Linux and a version of Windows as options. Only the Linux servers came with an uptime guarantee.

I worked on a service that had this exact issue; actually it was worse because it was a signed integer so rolled over in half the time.

Even more insidious, we never hit the issue until months into production because we had a pretty consistent two week release process. So the issue only came to light when development slowed down as the project became more stable.

Luckily deployments were staggered across datacenters so when the bug hit it didn’t happen to every server at once, but of course as always happens, when it did, the majority of the servers started hanging over the weekend while I was on call.

I worked at a company with a similar issue. The server is used to get patched and rebooted every month. Then the company decided to lay off huge chunks of operations staff. The software held up remarkably well, but we eventually discovered that software that was never designed to stay up for more than a month will leak memory like crazy if you leave it up for a long enough.

IIS used (still does?) to randomly kill its own processes at random because chances are, if its been running "long enough" its leaking memory anyway.

Is there a good way to test for bugs like this during development? Since it’s not particularly realistic to pause development for two months to see what a given build does after that long.

When you start development version of Windows CE or Windows Embedded (don’t remember exactly which one) tick count timer wasn’t set to zero but max dword minus number of milliseconds in two minutes. So counter overflows in 2 minutes after system start. This is very smart way to catch such bugs.

The same way you test for leap-year bugs, write unit-tested code that tests all your boundary conditions.

Hmm that doesn't seem to be the same thing.. it's easy to test for leap-year, that's a simple case you can either test in the code or literally change your computer clock and let it rollover to see what happens.

Very different than something which only happens after 50 days because obviously in small intervals it's not a noticeable bug.

The 50 days is arbitrary - the owners of GetTickTime should have written tests to handle what happens when the tick value rolls over. (Maybe they did, and allowing it to roll over was the desired behavior.)

But more to the point - the consumers of GetTickTime should have also had tests to verify that the consuming code appropriately handles time rollovers.

Granted, this was Win95 so I'll cut them some slack, but that should be standard procedure for any code that consumes an incrementing counter API. First question should always be - 'how will this code handle a rollover value'?

Our company does soak/longevity testing. Typically we'll take a build and let it run for a few weeks during development to find memory leaks. Then the final release enters our longevity lab where we put it under some amount of load and let it run until the device or VM is retired. It won't necessarily find the bug before final release, we could either recall that version or submit an immediate maintenance version that fixes the issue once discovered.

Software will always have bugs, we've chosen to approach it as "find the bugs before the customer" rather than "stomp out every possible bug". Because you are right, we don't get 2 months to pause and fix bugs, and also customers will always do some crazy configuration or workload you didn't have a test case for.

That's pretty cool. What industry are you in that y'all are willing to invest resources for such an exercise?

Back in 2005 I had encountered this bug or something similar. It was in Windows CE, the embedded version of Windows.

For me, it failed at around 25 day (49.7/2?). I believe the documentation at the time [since fixed] might have had GetTickCount() returning an integer instead of an unsigned integer. I had a devil of a time tracking it down!

unfortunately recreated this issue recently keeping track of freertos current tick on a 32 bit embedded system :)

Yeah, that happens. Here is an example of an actual bug opened a few years ago on our system. Dev team spend some time tracking the issue and found out incorrect variable type used for sessions, it was unsigned long used for milliseconds, like in the OP. We were joking that we need to allocate 50 days to verify the fix :) . https://imgur.com/a/HS3fcTj

49.7 days? I wish I could get more than 7 out of my MacOS Catalina laptops before I get a gray screen.

If you think this is cool, wait'll you see what un-upgraded UNIX-derived systems do on Jan 19, 2038 at 03:14:07Z.

Update: new kernel are on top of this problem. https://lkml.org/lkml/2020/1/29/355?anz=web It's the old systems that will suffer. Hopefully only soda machines and not avionics or intravenous infusion pumps.

In fairness, I have hit this exact same problem on Linux

This bug reminds me of the plot of the 1996 Japanese crime story "Subete ga F ni Naru" (The Perfect Insider).

the story that makes you love an incestuous homicidal genius.

I thought the anime really glossed over that story terribly.

From what thine is this? It says "There IS a bug ... NOW patched" Should make no sense anyway but I see no time attached to this.

The cited MS KB has been archived starting November 9, 2004:

https://web.archive.org/web/20041109020858/http://support.mi...

It is dated August 2004 (it is revision 3.2) but the patch (for Windows 98) has a date:

Release Date: Jun-04-1999

There's a CNN article about it from 1999 too: http://www.cnn.com/TECH/computing/9903/05/win98bugfix.idg/

I underlying "bug" [GetTickCount()] is still a danger/pot hole that you need to watch out for. I think it is timeless, especially for those who don't have scars from encountering it before.

In one of Jon Bentley's "Pearls" books is a chapter of rules of thumb. One states that "Pi seconds is a nanocentury."

Reply on news.ycombinator.com