Hacker News Reader

Comment by tominous | Hacker News Reader

I worked on an HSM system (hybrid disk/tape archival) which suddenly started having lots of I/O errors writing to tape. We tried new media. We tried new drives. We double-checked cables and SFPs. No luck.

Finally we tracked down the issue: when the contents of a particular file were archived to tape, the tape drive crashed. I suspect it was a tape firmware issue, maybe to do with the native compression.

The workaround was to mark that particular file as "not to be archived" and we stopped having media and drive errors.

View on HN

Ah, yes, the "best", as in fastest by far with highest quality results OCR software back in the '90s was owned by a company that was rumored to have pissed off their core technical team, who left and only occasionally deigned to do consulting for them.

So they had a wonderful core which was wrapped in baroque APIs, but the real problem was that core wasn't entirely wonderful, occasionally when you presented it a "Death TIFF" as we called the images for their file type, it would reliably crash. Software or firmware versions of the code (they had a hardware accelerated box with one or more Intel RISC chips), on the PC platform at least, e.g. Windows 3.x using a DOS box, this would entirely lock up the machine.

To get around this for a client that had 500,000 images to OCR on a tight deadline for a legal case (and this was the golden area of legal document imaging, back then lawyers would pay 50 cents per OCRed page, because a full text search could e.g. impeach a witness on the stand in real time), I created a system where the PC would always be printing out asterisks if it was OCRing pages. That allowed an operator to tour the machines and easily see when he had to manually reboot one stuck on a Death TIFF, after which my software would recognize what had happened and continue with the next image.

That's a quality hack right there

Yep, this is by some margin the hackiest thing I've ever done in my career. If I'd been doing it on Sun hardware, though, I would have been able to include power cycling hardware for the accelerator box.

I bet I worked with the same company when I was with the government. We had a subcontractor who'd been hired to digitize something like 200 million paper records (they made it about 50 million in before we ran out of funding). But a small fraction of the TIFF files they generated wouldn't work with any of the tools we had on hand.

It turned out that Windows 98 shipped with an Imaging program (Licensed by MS, not written by them) which predated the standardization of the JPEG-in-TIFF subformat, but they'd basically guessed at how it would work and shipped that. The final spec (and the version of JPEG-in-TIFF nearly everyone else implemented) ended up being different. So basically nothing could read it.

We ended up calling them up every time a customer found one of these files and having them print out that image on one of their windows 98 machines, and scan the printout back in using one of the newer machines. Sure, we lost some quality, but at least the customers could access the data now.

For a time reference, these broken images were still showing up in newly scanned documents in 2011 (when we stopped working with them due to massive fraud), so they must have been using their Win98 scanner systems even then.

No, to the best we could determine, and we had a guy who liked to get into the weeds of CCITT Group 3 and 4 compression, it was the raw images themselves, and there was nothing wrong with them, some just tickled a bug. If I remember correctly, their API required stripping off the header and presenting the OCR code with some metadata and the compressed image. It's been way too long for me to remember the details, except that it was fairly obnoxious to interface to, I couldn't just hand it a TIFF in some way (helped us VARs really "add value" and earn our keep :-).

We were producing our own TIFF files using our own software that drove monster Kodak ImageLink scanners (software I in fact took over, redid the SCSI driver of, and eventually did a clean rewrite of the engine on Sun workstations), so the images and their compression came straight from Kodak, and going further, I don't recall those 600 pound beasts ever screwing up at that level.

And this was way before Windows 98, it was Windows 3.0 or by then 3.1, like in 1992, Windows was utterly naive about document image files. Which I can see was a blessing (although maybe it was losing quality, I'd long switched to NT by the time 98 came out).

We also had weird CCITT Group 4 issues, because of someone trying to be extra smart and convert TIFF to PDF without a recompress (PDF supports Group 4 compression too, so you can turn a Group4 TIFF into a Group4 PDF by just swapping the header!)

I didn't mean it was definitely the same company, just a similarly annoying TIFF issue.

In a similar vein, I was messing around with Windows filenames. Using the extended path syntax, it's possible to use reserved words in a windows file name (com1.txt in this case)[0]. This breaks most tools that use the Win32 API (explorer, notepad, most COM components, IIS, etc). I showed everyone around the office, and laughs were had.

Fast forward a bit and a new backup system is being put in place. But it keeps breaking, but only on this box. While researching the issue, explorer keeps breaking when doing searches and third party search tools keep breaking.

Took me a little bit to remember what I'd done and fix it.

[0] - https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

Gosh, this takes me back. There was some special filename you could give a directory on Win98SE which would result in it being truly hidden, but the contents still accessible via some arcane workaround. I made such a directory on my first computer, then forgot about it, and--as in your story--eventually remembered it when trying to back up the filesystem.

Does anybody know the nature of the hidden directory hack I've referred to?

We used to make directories with just the character 255 (a hard space) as the filename. It would totally mess up File Manager in Win 3.1 (the tree would collapse when you touched the entry) and behaved very oddly in Win95/98 if I remember correctly. Maybe that is it?

Yes, I think you got it. :) https://www.dslreports.com/forum/r5309449-W98-Alt-255-phenom...

A friend showed me a trick where he typed a couple of assembly instructions in debug.com to change the DOS AUX device to something like BUX. That allowed creating or accessing an AUX directory. The converse assembly instructions would restore the AUX device and completely hide the AUX directory.

My high school (back in 2000) had a visit from a german gentleman who uploaded porn to their public ftp(!?).

When we (another student and I, also a student) tipped the Teacher-cum-admin off, the folders masqueraded as a printer in the NT file explorer. They couldn't delete them.

We recommended that they wiped the machine and disabled public ftp. (weren't that big of an issue as it was mostly a print server).

Reply on news.ycombinator.com