Repairing the card reader for a 1960s mainframe: cams, relays and a clutch | Hacker News Reader

Repairing the card reader for a 1960s mainframe: cams, relays and a clutch

righto.com

140 pointszdw65 comments

Heh, that very much reminds me of fixing an industrial sewing machine. This machine was made by Naumann, an East German absolutely rock solid piece of engineering.

The mystery was that one day to another it stared eating needles, and in some cases spat out the broken pieces at high velocity in all directions.

The good news is that you can spin a sewing machine very slowly by hand to see what's going on and while doing so I saw that - obviously - the needle was going down into the spool chamber at precisely the wrong moment, with the spool chamber still being closed. It's a very intricate mechanism that passes the needle thread under the spool using some black magic to pretend the mechanism comes apart while it does the move. Timing is super precise.

Like in the card reader example the origin of the problem was a part that had slipped on shaft. Apparently a bunch of thread had gotten stuck solid enough that it had wedged one of the cams hard enough that the shaft inside it had continued to move.

Even after finding the cause it still took forever to figure out the right setting (no markings on the shaft, no service manual for that particular machine indicating the right angle) but eventually I got it working and it still works today.

I love those old fashioned pieces of machinery, if a modern day appliance would break there is a good chance you would not be able to do anything about it, but this old stuff just takes some patience and basic observation skills and you can keep it going.

I spent a while learning about sewing machine timing and repair a couple of years ago after buying a very nice vintage Elna at a thrift store for $20. Turns out that the way most of the timing is set is based on the height of the needle when the shuttle mechanism underneath is in a particular place. You loosen the set screw on the collar that connects the upper and lower mechanisms, set the needle to the height of a gauge block from the manufacturer and then rotate the lower mechanism until something lines up, then tighten the set screw. Those blocks are almost all long gone for anything that's still mechanical, and the modern machines are all electronically timed with stepper motors in the top and bottom. Fascinating stuff if you dig into it.

That makes me want to go inside my older machine and mark the current positions, say with a sharpy or dye. Mine had the main plastic timing pulley break in half and is un-obtainium. So I epoxied the pieces together, while drilling holes and embedding steel wire reinforcement. Works like a champ, though I go the feeling it was built with planned obsolescence, as the rest of it was built like a tank.

Sometimes such a plastic parts sole function is to break to avoid major damage to the rest of the machine. If you make it stronger and the same conditions occur something else might break or bend that is a lot harder to replace or repair.

You'll see the same in farming equipment, there are usually one or two sacrificial bolts in there that will be happy to snap if the tool gets wedged, rather than that the tool snaps.

I get the sacrificial analogy, but in fact, the larger problem is IF the cam does fail, will it allow interference that will lock up at speed and potentially destroy the highly powered but delicate mechanisms, as you find destroys engine valvetrains, mainframe printers, among others. And this is what it did; when it failed everything locked, but luckily nothing was destroyed because otherwise built like a tank. The alternate explanation is that it is simply a very complicated cam device with many extreme contours, expensive to fabricate from anything but plastic injection, with high force followers that cause it to fail over time, leading to a trip to the sewing machine repairman. edit: I called it a pulley, but it in fact was a complicated cam.

That's a good use for 3D printing, assuming someone knows and can publish the appropriate measurements for the gauge block.

The problem is that no one (myself included) wants to do the work for the measurement. You need to retime the machine in front of you, and once you do, you don't really have any need to make up the gauge... Also, this is probably going to need to be more accurate than I understand 3D printing to generally be. I probably don't need to use my set of Starrett Grade A gauge blocks, but I probably want more accurate than your average ruler.

I have a desktop, sub-$800 printer and can print to 0.1 mm thickness without extraordinary measures and using inexpensive calipers, you can measure down to below 0.01 mm. (Thereby getting potentially more than native precision by repeated attempts.)

Thermal expansion of the plastics being different than the machine is a small issue, which could be solved by measuring, calibrating, and adjusting all at a given temperature.

Your point about limited motivation is spot-on, of course.

I took me a long time to figure out that under my magnetically attached spool holder/shuttle junk had built up allowing it to jump off the magnet, but only at high rpm, breaking the needle. BTW, when I went to school, you had to take Home Econ to graduate, and I learned to use sewing machines, utilizing sewing patterns and common tools like tracing paper, with a final project, as well as demonstrate many hand stitches. I wished my children would have had this opportunity.

Buy a cheap second hand (old!) sewing machine and you can give your kids that opportunity at home.

I've been on a project for a couple of months where I sort of re-play the whole industrial revolution and the electronics age one device at the time. The local goodwill store has a good customer in me.

Oh, they learned some of it (now adults), just not at the level I was exposed to - and that is it really, always competing priorities.

The prices of modern appliances have come down due to really intricate miniaturization of many components, which enables larger scale manufacturing and cheaper supply chains.

Because of the prices being low, it's not really worth it to repair consumer goods anymore. And if it's worth it in some cases (iPhone), it can only be done by a specialist due to the extreme miniaturization.

I've discovered that modern appliances are repairable, thanks to the internet. There are people that have been making a business based on supplying parts (for example for made-in-China GE dishwashers) and videos on how to install them.

I think it's more accurate to say that first world labor costs are such that it's not usually worth it to pay someone to fix your devices these days, but it's becoming easier to DIY.

I also fixed an old laptop based on mail order parts and a youtube video. It really gave me a sense of accomplishment because I turned to programming computers very young partly out of ineptness with mechanical devices.

Yeah, the Internet is a godsend to people who repair their own appliances. For the most part you just have to figure out what's wrong (most appliances are pretty simple, so this isn't very hard), then look at the part for a model number and punch it into Google. Failing that you can enter the model number of the appliance and often get a parts list. The part you get usually won't be an exact match, but it should work, although you may have to fabricobble a bracket if the new part isn't the same size/shape or has mounting holes in a different location. This is pretty rare though, usually it's just bolting the part on.

This is also when you discover that even though Frigidare may sell dozens or hundreds of different fridge models at wildly different price points over the years, most of them use basically the same innards.

The key for someone like me is a site with videos for every little part for the exact model of appliance. And troubleshooting information, and possible causes of a symptom ranked by customer experience %.

There have always been DIYers, but I cannot do much of anything with just a service manual or parts list.

Sewing machines are hard to miniaturize. I'll give you that the newer crop are cheaper, but they will also only last a fraction of the time, that decrease in price has also come at the expense of quality.

Not that that matters as much any more, people tend to buy clothing rather than make clothing, this particular machine was a wedding gift from my grandfather (a tailor) to my mother and it was meant to last a lifetime, so that the family would have clothes to wear. Making clothing was so common nobody thought anything special of the skill involved, everybody could do it!

But if you tried to seriously keep a family of four clothed today based on fabric bought and a modern day sewing machine you'd be buying new sewing machines ever few years.

Modern sewing machines are cheaper because the intricately machined cams and drive mechanisms that connected the upper mechanism that moves the needle to the lower mechanism that moves the bobbin are no longer used. They're replaced by a pair of stepper motors that are driven by a microcontroller. Instead of having collars and set screws to adjust timing, it's all solidly built and ruined the minute the stepper motor slips or a tiny plastic part breaks and something no longer actuates. They're unrepairable because you'd have to get the actual OEM somewhere in China to release the source code and put a jtag pinout on their boards rather than getting the chips loaded at the factory where the main board is made. That's the difference. Same reason that cam machines have been replaced with vertical machining centers -- CNC is just waaaay cheaper.

Conversely, back in the days if you had any kind of a significant IBM mainframe installation you had an IBM technician if not a team of them on an on-site contract to keep the thing running. The idea of an unattended data center, filled with computers that can run reliably for years untouched, was unthinkable then.

When I was still working with mainframes we used the IBM diagnostics on a Sperry machine (it could emulate the IBM instruction set), they were so much better than the Sperry ones that hardware faults would be isolated much quicker.

One of the best hackers I ever worked with (a guy called Paul Poelenije, unfortunately deceased) had figured out how to load the microcode store and to pull this trick, I'm pretty sure that both IBM and Sperry would have been horrified at it in equal measures. For an encore, he wrote a completely new filesystem in assembler, and used it in production for the bank we both worked for. How he got away with that I'll never know but somehow he managed to get the blessing for his pet projects from the manager of the systems programming division to who I'm going to be eternally indebted because he had a big say in me coming to work for the IT department instead of the mailroom.

Those Sperry's were massive machines, the two mainframes (primary and backup, or production and test depending on what we were up to) + peripherals took up all of one floor of a very large office building and there were always at least several people on duty to keep them running. This was mid 1980's and punch cards were still being used with some regularity.

+1 for even knowing the word "Sperry."

I named my dog Sperry simply because nobody in my company had ever heard of it, and I didn't want the name to die.

A long time ago, I discovered that a nearby neighbor had a beautiful Mercedes limo from the 50's in his garage. I asked him about it, and he said he had restored it. I asked him how he dealt with acquiring parts that were no longer available.

He replied: "Oh, I made them."

Me: "What...."

So he showed me his full sized basement, which was a fully equipped machine shop. He'd built working steam locomotives that were several feet long, and other mechanical models, all working, all from scratch.

I was amazed. I asked him what the Mercedes was worth, and he replied, and I remarked that it couldn't be worth all his labor he put into machine work. He agreed, and said it was his hobby and evidently made him happy.

These days, of course, you can just make a drawing of what you need and send it to one of those online machine shops.

Ahhh, that's the dream. I just let a friend use my mill to bore out and sleeve the cylinders in the brake calipers for his vintage Corvette rebuild.

On a side note, that's the sort of thing you generally don't know about a neighbor -- that they have a complete machine shop in their basement. I wonder occasionally what miraculous things people I don't know well do in their leisure time.

Mercedes-Benz "Classic Center" still has parts for all their cars, at least the post-war cars.

Not to say that it might not be cheaper to make your own in some cases if you already have the means and skill to do it.

My family restored (to good driver condition) a '59 220S (just sold); our experience was that not every minor cosmetic or interior part was available at Classic Center. Mechanical and functional part availability is excellent, pricing reasonable, and the staff there is top-notch, but they don't have everything.

Maybe they do now. That fellow had restored his in the 1970's. Or at least with the internet it is a lot easier to find someone stocking old parts.

Jay Leno's Garage employs machinists to make unavailable parts.

Most car companies start to shut off making parts for cars older than 10 years.

I've heard that it's not uncommon for blue-water sailors with small machine shops on their boat. One story was about a guy making a new cylinder for the diesel engine on his lathe while being tossed around the cabin in a storm. That's a case where you need it right now and can't order it online :)

Tell me more about those online machine shops. Where can I get metal parts machined?

google "online machine shop"

(I haven't used them myself and have no recommendations.)

Fantastic as always Ken. I spent years of my life repairing IBM mainframes and I love these info drops.

I guess its part of human nature to assume that our predecessors were dumb peasants (how did they build those pyramids?) but IBM had some incredibly sophisticated and elegant electromechanical, pneumatic and hydraulic systems in those days.

The worst beast I used to work on was the 3890 check sorter https://en.wikipedia.org/wiki/IBM_3890. This monster filled a (large) room, and when it was down, stress points went through the roof as bank's check sorting and money clearance ground to a halt. There's nothing like having a furious CEO leaning over your shoulder shouting at you while you're elbow deep in a dusty noisy furiously spinning machine to raise your blood pressure.

> The Computer History Museum in Mountain View runs demonstrations of the IBM 1401 on Wednesdays and Saturdays so check it out if you're in the area; the demo schedule is here: http://www.computerhistory.org/hours/

Regarding the IBM 1401 (to borrow from "Ferris Bueller's Day Off") It is so choice. If you have the means, I highly recommend seeing it in person.

I've had the pleasure of seeing several presentations and even punching cards to be run as part of a demo program. The machine is impressive to see in person and the volunteers who run the tours are a lot of fun to talk to!

Agreed - if computer tech is your thing it's a really good place to visit.

I was roped in to one of the Babbage Engine demos a few years back...I turned up about 10 mins early and started to talk to the presenters, who were really nice guys. When I volunteered that I'd also seen the other replica Engine in the Science Museum in London, the chaps were proud to point out that 'theirs' could be operated and was not behind glass. I was then asked if they could refer to me at a suitable point to mention those facts and give my impressions on the presentation, which I was happy to do!

Icelandic musician Jóhann Jóhannsson composed an album of music created with sounds created by the IBM 1401. This particular track also includes what seems like an audio recording of a technician going over the machine's specifications and some basic troubleshooting: https://www.youtube.com/watch?v=KPFAkBCizr8

More information here: https://web.archive.org/web/20120307215225/http://www.ausers...

>In 1964, a computer - the IBM 1401 Data Processing System - arrived in Iceland, one of the very first computers to be imported into the country. The 1401 has been called the "Model T" of the computer industry - the first affordable, mass produced digital business computer. The chief maintenance engineer for this machine was Jóhann Gunnarsson, my father. A keen musician, he learned of an obscure method of making music on this computer - a purpose for which this business machine was not at all designed. The method was simple. The computer's memory emitted strong electromagnetic waves and by programming the memory in a certain way and by placing a radio receiver next to it, melodies could be coaxed out - captured by the receiver as a delicate, melancholy sine-wave tone.

When I was a grad student back in the late 70's, I worked in the campus computer center where all the undergrads had to use Hollerith cards and only the grad students got to use CRT terminals. There was one card reading machine that was known to be flakey and would mis-punch the letter E occasionally and I used to love it when they would come up to me all frustrated that their program would not run and I would pull the offending card from their deck and hold it up to the light and pretend like I could sight read Hollerith and tell them they had a mis-punched card. They all thought I was a god... until someone would clue them in not to use old reader #11.

That's such an IBM design. IBM mechanical designers were used to designing systems with continuously running shafts and electromagnetic clutches, because that's how tabulating machines worked.

Early card readers were unreasonably complicated. Then came Documation.[1] They solved the problem of card feeding. There's a air jet system which spaces out the cards, and a vacuum picker which pulls the bottom card off the stack and feeds it into feed rollers. The card path is straight and simple, card picking is not highly sensitive to card thickness, and the machine is far simpler than the 1402. Documation soon owned the card reader business.

[1] https://www.youtube.com/watch?v=qu55b0GpgE8

It's humbling to thing that today, doing the same thing (if for some reason we'd like to use punched cards) we'd do the same thing with a $10 camera, a $35 Raspberry Pi and some trivial OCR-like software. Heck, even an ESP8266 could process images that simple.

The machine in the article does 13 cards per second. A cobbled together OCR machine probably isn't going to match that.

Exactly! Being able to reliably read 13 cards a second from a stack is quite a feat, requiring precision components such as picker knives and jogglers moving at high speeds. (I am not making up the joggler; it uh joggles cards into alignment.) Optical reading would avoid the metal brushes, which get out of alignment or make intermittent contact, but that's not the hard part.

Yesterday's "Why Paper Jams Persist" in the New Yorker is also relevant, a look at why its difficult to move paper through a printer reliably. Punch cards, being made of paper, have the same problems but worse - they are weakened by the holes, and mistreated by the public. (Older readers may recall "Do not fold, spindle or mutilate" printed on punch card utility bills and such.)

https://www.newyorker.com/magazine/2018/02/12/why-paper-jams...

  reliably read 13 cards a second

... and with considerable tolerance for bent or frayed cards.

As late as the late 80s, Lockheed used IBM cards for time cards (for everyone, hourly or salaried). Lots of handling, lots of abuse, yet over 99% read ok on first try. That's over 14,000 in one job (program) every week for years on end.

The majority of the machine seems to be the heavily engineered card feeder with sensors on every conceivable point of failure. That would still be a hard problem today, especially if you're asking it to read ancient and potentially damaged cards.

You could probably get something like a 10+ year old document scanner like a Scansnap 500 cheap. Those can hit 18ppm.

edit: I realize this isn't 13pps, but it's not bad.

Any old 1080p video sensor will do 60fps

Yes, but I'm guessing the reliable mechanical feeding of the cards is the harder task than the actual image capture.

In 1985, my high school inherited an old PDP mini computer and we had to use optical cards to write BASIC programs in my Data Processing class. It was kind of a trip, since most of the computer nerds had Apples, Commodores or whatnot at home.

The code and debug cycle was to mark your cards with a Sharpie, run them in the card reader, and then look at the printed output for bugs. If you were unlucky enough to have bugs AND drop your cards on the floor (I remember it happening to a couple of people in my class), you had to waste a bunch of time re-sorting your cards before you made the trip back to the card reader. Good times.

You should take that sharpie and draw a diagonal line on the side of the deck, then it's easier to get them back in the right place if you drop them.

Hence columns 73-80 being used for a sequence number.

There's interesting history behind why columns 73-80 were used for a sequence number. FORTRAN was developed on the IBM 704, which was a 36-bit machine. The hardware read punch cards "sideways", storing each row of the card in two words. The consequence was that you could only read 72 columns of a card. That left columns 73-80 unusable for programming. But if you punched a sequence number in these columns, a dropped deck could be easily sorted by a high-speed card sorter.

Card sorters were an interesting technology, going back to the 1920s, way before computers. They sorted cards on one column at a time using radix sort, and were electro-mechanical. I wrote about them here: http://www.righto.com/2016/05/inside-card-sorters-1920s-data...

> But if you punched a sequence number in these columns, a dropped deck could be easily sorted by a high-speed card sorter.

I suppose heaven help you if you wanted to insert a statement between cards 37 and 38 :-)

That's why you numbered your cards with 2, 4 or even 8 spaces between each subsequent card number.

That sort of forethought came from hard won experience ;)

Similar to how BASIC line numbers are multiples of ten - allows for insertion.

If you're going to dig around in the source code for a 1970's BASIC interpreter one of the more interesting commands to disassemble is 'RENUM'. It tells a lot about how the language stores its intermediate structure (not quite a bytecode, but it is highly tokenized).

RENUM takes a program that has had insertions done to it and renumbers the lines so they are multiples of 10 again. For added complexity and to avoid overwriting your carefully laid out library calls with high line number subroutines for easy memorization you could limit the range within which the RENUM would operate.

Typical use: RENUM 0,1000,10 -> renumber all the lines from 0 to 1000 with a step of 10.

Unfortunately, RENUM doesn't work for card decks.

Well, with enough tape and patience :)

  mark your cards with a Sharpie

We used Mark Sense cards marked with a #2 pencil for the PDP 11 in high school.

In the UK we used coding sheets and got the results back next week.

CuriousMarc (I think, not sure if typed correctly) has a lot of behind the scenes videos on YouTube for that vintage computer stuff, mostly at the Computer History Museum where they work on the IBM 1401 and the Xerox Palo Alto.

Some of the repairs are for the tape drives, some are for the CPU itself, it's all really interesting to see.

CuriousMarc's YouTube channel is https://www.youtube.com/user/mverdiell and has lots of videos of the IBM 1401 as well as the Xerox Alto restoration, HP hardware, and other interesting things.

Cool stuff, I'm watching [The IBM 1401 compiles and runs FORTRAN II](https://www.youtube.com/watch?v=uFQ3sajIdaM) and seeing that alternating green and white printer paper brought back some old memories of when dot-matrix printer paper was used as scratch paper during school days.

Impressive to even see its still for sale https://www.google.com/shopping/product/13500800368299416699

I like how they also have a lot of stuff on custom hardware, like FPGA boards to read the diablo disk packs and the 1401 tape emulator.

I love articles like these, even though I wasn't alive when these machines became widely used they are fascinating to read about.

Unfortunately I live nowhere near a museum with a functioning mainframe, but I would love to see one in action someday.

A good write-up and read, as ever, Ken.

I've a punch card from that machine (somewhere!) - picked up the last time I visited the museum.

I used to fly over from the UK a few times a year to visit corporate HQ in San Jose, and I used to enjoy taking the VTA light rail to Mountain View at the weekend and walking to the museum.

Sadly, I have now changed jobs so the chances of coming over again are very low - unless you need to hire a 'specialist' to hold your multimeter!?

We can always use a specialist multimeter assistant, but the pay is low :-)

OK, I'm in. If pay is low, I may have to line up a side gig at somewhere nearby - know any local business that do computer-oriented stuff!?

Also be sure to find the link to the schematic, which is kind of buried in a footnote. 59 page PDF.

http://bitsavers.org/pdf/ibm/1402/1402_Reader-Punch_Wiring_D...

Ken was recently in the Unnamed Reverse Engineering podcast and it was very interesting too: http://reverseengineering.libsyn.com/005-circuits-that-go-to...

Were things like servos and stepper motors things that came after this era? This seems... awfully mechanical for what it’s doing.

Reply on news.ycombinator.com