Hacker News Reader

Comment by ajford | Hacker News Reader

When I was still working at Arecibo Observatory, there was definitely a couple systems running Win95 that were considered data critical. And they were still running on ~486 hardware IIRC.

They hadn't been replaced as they were running some custom ISA interface boards developed by a research group in the 90s, and the community was still using the data output by the machine. And since it had been trucking along for ~20 years, convincing them they needed to develop a replacement was hard.

When I left, it was still there, chugging along. And once a day, a tech would wander in with a floppy and copy some data off and wander back to the control room where a USB floppy drive was attached to a workstation specifically to read this floppy and let the tech copy the data files off to a network location.

View on HN

Sounds like their system works. If that 486 hardware lasted 20 years already, seems like there's a pretty good chance it'll outlast most of the modern stuff it would be replaced with.

I suppose its fine as long as its not connected to the internet.

Do you suspect there are any 0day/worms still targeting Windows 95 systems?

I wonder if an unpatched Win95 box is actually safer than an unpatched modern Windows box.

>Do you suspect there are any 0day/worms still targeting Windows 95 systems?

You'd be surprised how much malware is "legacy" 32-bit and works without issue on WinXP. A lot of malware even shies away from Unicode API's and specifically calls out the *A Win32 API functions.

Considering the Win32 API is backwards compatible to a fault, I don't see why modern malware couldn't absolutely wreck a Win9x box.

Sure – once the malware lands. But I doubt anyone is going after any Win95 vulnerabilities, so how does it get on the box to start chomping?

Loaded from the boot sector of any floppy that happens to be present when the system is restarted, unless the boot order has been changed from factory defaults and the (possibly very old) coin cell powering NVRAM holds sufficient charge to retain these changes when (if) the system is ever powered off.

In general, attacks (e.g. most ransomware campaigns) are not automated malware spreading on its own, they involve people. If it's accessible on a network (which is the big 'if'), then it's obvious on any scan that it's a Win95 machine, and then you can try to look up a vulnerability for any exposed service. For example, if file sharing is running, then all the SMB vulnerabilities (e.g. EternalBlue and friends) apply also to Win95 systems, and I'm not sure if Windows 95 had a patch made, backported and issued for them.

Yes, but Windows XP was in widespread use until recently and is still in use in a lot of places. I don't know where "netmarketshare.com" takes their user statistics (Google just used them when i searched for that) but it reports "1.26%" for Windows XP, which i think is still a large number. There are still so many users of Windows XP that Microsoft released a critical security patch last year despite XP being official EOL for years now.

Windows XP (NT kernel) and 9x are totally different OSes, using totally different kernels. They share some API, but not really the interesting parts.

I work on software that still supports XP with only a little pain. Visual Studio 2019 will still install the necessary toolchain if you ask nicely. Targeting Windows 95/98 in 2020 sounds like a nightmare.

> Visual Studio 2019 will still install the necessary toolchain if you ask nicely

Could I get a link to how to do this and/or web search terms? Is this just something targeting visual C runtime 20xx or older?

Eh, no. Not by a chance. I have some old as heck games which don't even work even under W2k with w9x compat mode (it's hidden, you need a command to enable it), so a total nope under XP.

That floppy disk controller is the only way into the box and isn't nearly as vulnerable as a modern usb controller since there's no smarts!! The windows 95 system is going to be safer than a patched windows 10 on the network for their use case.

For the common-case en masse compromises, not a concern. For targeted attacks it can still be an issue. Stuxnet is an example of this where they compromised an air-gapped system. Stuxnet was obviously extremely sophisticated, technically and for the overall operation, but the principles still apply. Also people frequently misjudge how big of a target they may be & what their risk profile really is (not saying for this use-case specifically, just in general).

Win95 is running as root by default. Any process can open another's memory. It has the same registry entry as Windows NT for running a program on startup. The keylogging API is basically the same. The IShellFolder and IWebBrowser interfaces are identical, so as long as the malware didn't request specific classes or features that are outside the realm of Windows 95, it can probably do pretty well. Luckily, Internet Explorer 3 doesn't support TLS.

Except for the Ex functions. I wonder if a malware can be run under KernelEx.

IIRC, the machine was on an isolated equipment network with no external access for some time, but the network drivers would often cause some kind of conflict with the ISA interface card (and custom driver) so the network interface was disabled. So yeah, should be safe (and is likely no longer hardwired to the network).

Risk is hardware related, finding replacement parts. But yes, the software part seem to work fine.

It works yes, but they were running out of spare parts when I left. That system's parts were on frequent Ebay searches, as it was becoming quite fragile. I think it was on a 3rd power supply and a second mobo or proc, can't quite remember.

It was on a redundant on-line UPS as the last time it went completely off and was booted from cold, it ended up needing parts. So no, I'd say it's not likely to outlast modern replacements. Just had lots of support and organ transplants.

Systems like that ultimately end up being a resource drain, as unlike modern hardware it's not simply wandering over to a retailer and buying standard replacement parts, but sourcing used or NOS parts and hoping they work, often requiring far more time and effort by someone who could be doing some other important task.

The problem is that in open use observatories like Arecibo, a lot of equipment like that is put in place by third parties, but becomes our responsibility to maintain. But the science is important, so we maintain it as best we can.

Eventually, yes, it becomes a resource drain. If you want to know when, ask the finance people.

> If that 486 hardware lasted 20 years already, seems like there's a pretty good chance it'll outlast most of the modern stuff it would be replaced with.

Unlikely. Good quality 'modern stuff' has far better quality than what was available back then. Power supplies have gotten much better (and have more protections), as did motherboards with solid state capacitors. Older motherboards loved their electrolytic capacitors.

Processors had less protections. Thermal monitoring was at its infancy.

Memory sticks back then were very unreliable. As were hard drives.

Then there's just a matter of age. If it has lasted 20 years, chances are it won't last much longer.

It has probably lasted 20 years because it wasn't being powered on and off constantly. It most likely had stable power sources and - most important - a climate controlled environment with low amounts of dust.

True, but people have backported modern hardware to that era. You can take a modern power supply and drop it into that PC, I'm almost sure.

A 486 system was built before the capacitor plague. Quality electrolytics may not last forever, but can last a long time. Also, built before RoHS and the reduced longevity of poorly formulated or poorly applied lead-free solders that were common for many years.

Processors had less protections, and thermal monitoring was at its infancy because almost all the 486s wouldn't use enough power to cook themselves to death, even with passive heatsinks.

I don't know about memory sticks and hard drives from that period. It's quite possible they've replaced the hard drive with compact flash or something too, which should be pretty reliable given they apparently only make a floppy disk's worth of data every day.

I'd love to hear more about what "the capacitor plague" is

https://en.wikipedia.org/wiki/Capacitor_plague

It's why computers from the 80s can last until now without needing any recapping but computers from 2002 or so are ticking time bombs.

But not after 2007?

Well, for some years after 2007, you still have old stock bad capacitors floating around. And you're into lead-free solder problems.

Um.... All electrolytics eventually go bad as the electrolyte solution dries out eventually. Re-capping retro computers and radio gear from the 70s through the 90s is very much a thing.

That being said, electrolytics were somewhat less common than they are now. But the Capacitor Plague is not the only reason electronics are recapped, especially older/vintage/retro electronics.

Yeah, but the 70s-90s stuff may die /eventually/ but not pretty consistently after 2-3y, that was the point.

This! Got a used AMD-K5 PR133 in a big-tower a long time ago, integrated it into my home lab, and used it to experiment.

About 1 one year later it underwent my cleaning routine, vacuuming dust out, cleaning the fans, etc. I took the cpu-cooler off to check for bad thermal grease and discovered it had none at all! No pad either, someone just clipped that thing on, and instead of direct contact it had a small airgap which you could look through from the sides.

Unbelievable.

When I got it, it must have been in use for years already!

Modern components are run much closer to their limits, simply due to engineering optimisations.

A 486 doesn't even need a heatsink, it's passively cooled and perfectly fine with that, as software of the time (mainly DOS) didn't even "idle" the CPU --- it was drawing full power all the time.

That also means if you manage to boot DOS or similar era software on a modern machine (especially a laptop) that was designed to have the CPU idled for most of its life, it is likely to go into thermal throttling immediately.

Is that really true? I booted FreeDOS (had it installed for a while just for kicks) on my Thinkpad T420 (i5-2540M) and I didn't notice any heatup.

Yes and no. The 8086 had a HLT instruction that was meant to replace the busy loop waiting for an interrupt. It was soon understood that you could also save power by turning off the clock. I'm not sure if the "SL" versions of the 486 turned off the clock with the HLT instruction, but later versions did. Use of it was apparently introduced in MS-DOS 6.0, so FreeDOS, Windows 95 and OS/2 2.1(?) would have probably used the instruction to effectively idle the processor. Not sure if Windows 3.1 made use of it, but it would make sense as the basis of the system's event loop.

https://en.wikipedia.org/wiki/HLT_(x86_instruction)

I recently installed DOS 6.22 (and Windows 3.11) in a VM and it causes one core to sit at 100% all the time.

I don't know if the VM changes anything, but it definitely seems as though it does not halt while waiting for activity.

I can recall there were programs like "Waterfall" and "Rain" which were supposed to force Win9x to HLT when idle.

Besides what the sibling reply mentioned about capacitors and processors, WRT memory that machine has survivorship bias. The fact it's been running for thirty years tells you the RAM in the machine is good and is unlikely to have a problem barring damage. The same is true for all the other still working components.

After 20 years, might they be in territory beyond the far side of the 'bathtub curve'?

> Sounds like their system works. If that 486 hardware lasted 20 years already, seems like there's a pretty good chance it'll outlast most of the modern stuff it would be replaced with.

Surely it does make some sense, but think about the hardware suddenly breaking down and not finding what to replace it with. Even with spare parts it could be a pain as the old generation retires

Just get an industrial board which provides ISA-slots and has something like http://www.vortex86.com/ on it.

I actually was surprised to see these are made, as I considered getting one for a few projects. The ones I saw were pretty expensive with some being resold on ebay for just under a thousand dollars. Reading some reviews though it seems yoy have to be careful as not all the boards may support what you want. Apparently a good many don't support DMA.

I thought about this, but let it stand.

It can be tricky. I had to work with something like this a few years ago and managed. As I remember I had to get Scitech Display Doctor/Univbe from some abandonware sites to make use of the weird onboard VGA under Windown95. The ISA-card for the CNC-machine worked, it used DMA, among other things. It all worked, and was even more fluid to operate/program.

The cost was just over 400EUR. (for the board)

Anyways, it's industrial, the formfactor often does not fit existing cases and backpanels, but it's doable. More so than hunting ebay or similar for spares of dubious provenance.

Only exception would be if the "App" and the board/card were designed so sloppily that they really only run on exactly that same hardware.

This is due to the rule of antifragility.

https://en.wikipedia.org/wiki/Antifragility

I worked in an office with a similar setup that "worked", but it was very fragile. When the HDD started failing on the old 486 machine we scrambled to migrate as much as possible (including a few rewrites and ports of utility programs). We were lucky that our problem was the software not running on modern systems (for reasons), not a dependency on the hardware. Mostly we communicated with hardware systems via serial, so any computer with a serial port (or a USB-to-serial adapter, which are often unreliable) could be made to work if the software existed.

There’s a word for this concept, the ‘Lindy’ effect. https://en.wikipedia.org/wiki/Lindy_effect https://www.urbandictionary.com/define.php?term=Lindy

How are they doing backups of the software and data?

The entire disk (all 10s of GBs of it) was cloned and backed up as a disk image, and can be re-imaged whenever necessary.

The data is copied off daily via floppy (don't get me started...) and is backed up on local network shares. The specifics of that weren't in my need-to-know and was managed by the IT group.

The system was essentially stateless, so spinning it up from the image wasn't an issue, but the PC itself is essentially part of the "equipment", as it ran the custom drivers and it was specifically designed for that PC and a one-of-a-kind ISA card.

I wonder if it's even just as easy to just run some of that on VirtualBox / some hypervisor running Windows 95, that way they have access to modern replacement hardware.

But there’s also a fair chance something will break and you can’t buy spare parts for museum pieces like this.

Reading this... my first thought was "But newer x86 can.... oh right ISA..." then I googled for USB ISA slots... and they are apparently a thing. That doesn't mean someone wouldn't need to write a new driver but at least they could still use the cards.

There are all sorts of USB-to-Foo interfaces, the vast majority of which implement whatever Foo is just barely good enough to say it works with a straight face. Like the USB-Serial adapters...most of them work ok at 9600,8,N, but trying 5-bit with 1.5 stop bits (something a 'real' PC serial port can be persuaded to do) and good luck.

So, yeah, I'm sure that USB-ISA adapter probably works with that old 3Com ethernet card or that Soundblaster card that 'definitely' sounds better than anything else, but a custom card with custom software and probably unknown tolerance to variations from 'real' ISA? Good luck with that.

Oh probably and even if you did find one that works "well" you'd have to buy a ton of them to ensure you have backups. I'm well aware of the issues. But without an alternative or a grant to update the hardware what do you do? At some point the availability of period hardware with ISA slots is going to be prohibitively expensive.

You can still buy new manufacture motherboards with ISA slots and modern processors. Intel still (or until recently) made ISA bridge chips for modern Southbridge chips. And it's something that a smallish FPGA could handle. There is a whole ecosystem of companies that recreate old-style motherboards for exactly this scenario.

My brother's company does IT stuff for the manufacturing sector. Very conservative. I've worked with a number of his customers to source exactly this sort of thing. Years ago they asked my thoughts on an Italian made cloth dying machine the size of a bus with a DOS PC controller and a minimally documented ISA controller. A mid 7-figure US$ setup; the company that built it is long gone. We found a shop that supplied new motherboards, modern (at the time...P3/P4 era I think) with ISA slots the vendor guaranteed it would be every bit as slow and weird as a PC/AT. Migrated everrything over, the company bought a stack of motherboard spares new, and I think they're still using them.

Now...you want a challenge? Migrating the proprietary, mostly undocumented software, all in Italian (we aren't native speakers...though my high school Latin occasionally helped) from an old ESDI disk to something, anything, else. That was much more interesting.

Just like most USB-to-parallel adapters will only work with parallel port printers and not much else.

Many years ago I was using an MCA (multi channel analyzer) ISA card connected to a surface barrier detector to measure beta decay spectra. This stuff basically counts the number of pulses from the detector binning them according to the height of the signal. It had its own acquisition software meant for DOS (the manufacturer provided the sources as well).

The thing is you could operate it under Windows (98) in protected mode, but then you lost statistics. How come? Well, Windows scheduling meant that the MCA only took data while it was allotted time slices, as a result the dead time of the whole setup increased, and you couldn't tell by how much.

Under real mode DOS it took advantage of DOS being almost an RTOS and the program running as a single task and you could be pretty confident about your data.

I don't think using adapters with this sort of cards is a good idea. It's not a matter of drivers, sometimes the old stuff simply doesn't work well when plugged in newer systems because everything has changed and you need to redesign everything. If the old stuff does the job, the OS being unsupported only becomes an insurmountable problem for the folks that consider using an unsupported OS some sort of crime in itself. Well, there's usually more than just an abstract computer being used in such circumstances.

Exactly. I believe someone tried investigating a solution to use a modern system and either perform some kind of pass through to a Win95 VM or some such magic, and it ultimately came down to needing to re-implement either the driver or the software, and that is hard to do without funding and time. Or any good documentation on what the original function and behavior was.

Yeah my first thought was just running a 16bit OS on modern hardware (it will do it if you're careful and do the right things). But then the issue of it being an ISA card popped up... that's not really a solvable problem with modern hardware unless you can convince an IHV to make you a custom motherboard. Although apparently some exist ( https://www.ibase.com.tw/english/ProductDetail/EmbeddedCompu... ) probably for industrial applications. While I'm very sure you could get DOS running on that... I'm less sure of Windows 95. The google search did show other options that are plausible but I doubt many of them would arrive working as is without needing capacitor replacement at a minimum.

To save others a search: http://arstech.com/install/ecom-prodshow/usb2isar.html - about $145 each.

Currently I have a related task, a telescope is being operated using a set of old hardware (late 90s-early 00s) and same age software, due to current situation in case of lock-downs the observations are not performed, so the idea is to migrate everything into virtual machines with PCI forwarding, so that VNC/whatever other remote desktop software can allow remote operations. Does anyone have such an experience? What are the pitfalls? I plan use Xen or KVM.

I did something similar for an organisation that had a strange music licensing setup for pub jukebox machines. A significant portion of their business hinged on this one ancient DOS pc with custom hardware and a licensing dongle.

Ultimately it comes down to whether you can get the inputs and outputs you need replicated/attached to the virtual machine and and exact copy of the machine itself into virtual form - the latter challenge can include having to do risky stuff like physically removing the source HDD to plug it into various interface converters and then taking an image from it using an intermediate machine that can read it.

Depending on how your setup works you may run into issues with things like time precision within a VM - if high time accuracy is important in the work it’s doing you’ll need to put attention into ensuring VM time = real world time.

Some top of my head suggestions based on our experience:

- Use Proxmox. Even free version can go a long way (because backing up VMs periodically is a good thing and Proxmox can do it fine).

- If you're going to run Linux, run XFCE desktops with X2Go server. X2Go can work almost in any network condition.

- Test PCI forwarding extremely well before taking the plunge. Especially if you're going to virtualize specially designed equipment. If it works, it's magic. If it doesn't then you're in big trouble.

- If the PCI cards will be migrated from older equipment to newer one, test to see whether they work well with spread spectrum. If it doesn't make sure that the server you use allows disabling it.

The amount of questions I have about this would mean we’d be here for days but:

How long ago did you work at Arecibo?

What was it like?

What did you do there?

Did you ever... find.. anything that you can’t talk about? Blink twice if yes!!

If you're really curious, hit my profile and email me. Always happy to chat about AO and the fun stuff there.

I was there for about 3 years and left around 2016. It was one of the best and most interesting places I've worked at so far.

The people were great, the work was great, and I personally felt the work came with a huge feeling of accomplishment and made me feel like I was contributing to a better world by facilitating scientific progress.

I was part of the Electronics Dept. which was ultimately responsible for the scientific hardware, the signal paths, and everything between. There were some people in Physical that were more responsible for the electro-mechanical stuff like motors and such. I was primarily responsible for assisting with the analog signal path and was considered a Receiver Specialist, but the latter half of my time there was spent writing a lot of digital control stuff for newer projects as I was the only one on staff at the time who had the skills and time to take those on.

As for finding anything..... that's classified :p

We did manage to talk to an old NASA satellite from the late 70s. The ISEE-3/ICE reboot project. Some details on Wikipedia here: https://en.wikipedia.org/wiki/International_Cometary_Explore...

What did the ISA boards do?

Also the name sounded familiar, this was the radio telescope which incurred damage recently.

The ISA board was a custom interface to a number of GPS receivers and some IO chips for time sync calibration and tracking for Pulsar astronomy.

And yes, this is the one that recently suffered a cable failure that damaged the reflector.

Curious, how long ago was that? I visited last year and got a tour from Angel. Yea, they had a lot of old hardware.

When I left? Around 2016.

I'm almost certain that '95 machine is still sitting in the Clock Room to this day.

Man, I miss Angel, gotta shoot him an email sometime soon. Miss sitting around and chatting about ham radio with him!

Reply on news.ycombinator.com