The Forgotten Operating System That Keeps the NYC Subway System Alive (2019)
vice.com
vice.com
You might think this is a clever take but it's really not far from the truth. They NEED a revamp on a modern tech stack so they can use the trains, electronics, and technology trains in the rest of the world run on.
In other words: the things that are currently falling apart (the physical relays, for example) are almost completely disconnected from fare collection (which is what the article’s about). The latter runs pretty well; replacing it with a modern tech stack rather than the current incremental approach wouldn’t have many benefits.
Arguably, the most advanced "tech stack" for moving trains around rail systems today (at scale, with the highest safety rating) is based around the Sicas ECC electronic interlocking by Siemens.
However, this article is merely about the technology used for collection of fares for the NYC metro rail system.
> Most subway services cannot significantly increase their frequencies during rush hours, except for the 1, G, J/Z, L, and M trains (the L service already is automated with CBTC).
> However, even without CBTC, the system is currently retrofitted to operate at frequencies of up to 60 trains per hour (tph) on the IND Queens Boulevard Line (30 tph on each of the local and express pairs of tracks made possible by the Jamaica–179th Street terminal, which has four sidings past the terminal for each set of tracks) and 33 tph on the IRT Flushing Line.
https://en.wikipedia.org/wiki/Signaling_of_the_New_York_City...
Imagine if we required $ billions in public resources to run other mundane software projects people came to rely on.
IIRC, they eventually dumped DAE, but had to deliver something before CORBA (or DSOM) was a thing.
https://web.archive.org/web/20071012190431/http://pages.prod...
The power and energy efficiency of today's computers is wasted on commercial surveillance and draining advertising budgets. Let us pray that someone discovers a more effective means of advertising than the computer. The 2018 Netflix series "Maniac" seemed to hint at this alternate future.
Put together, the daily weekday ridership is probably closer to ~14M, pre-COVID. And that's not including some of the knock-on computational costs, like the radio network that links the MTA's buses or the in-system transfer rules.
It turns out, in my opinion, we solved most problems in software engineering decades ago. All this high flying non-sense, web3 petabyte scale cloud powered type safe microservices, is pointless navel gazing. Agile is pointless navel gazing at least in it's current incarnation. Our entire industry has been mired in bullshit that hamstrings us.
There's also a tremendous difference in developers. Developers even two decades ago were near top of their game because they had to be. They didn't have kubernetes-docker-CI/CD-SRE-devops to protect them. The result? Far better, far more reliable code.
Any idiot can write software. Changing languages won't save you. A good programmer can make any language into a good language - and as it turns out 30 years ago we did it right. If they're only encountering scaling issues now that isn't a knock against them. It goes to show just how well they took "primitive" tools and made a system that simply works. Asking a modern dev team to produce a system that simply works often requires several months of planning, two full time product managers, a tech lead, several weeks of architecting, and approval (in blood) from the CTO who somehow has to involve themselves in literally anything and everything. Let's not forget we have to get it spun up with terraform so it's in the cloud and that'll take a request to SRE and possibly even dev-ops to get done. By the time the developers even get to write code, the deadline already arrived! We are comical facsimiles of our fore-developers and frankly should feel ashamed.
22 years ago, I was working on software to run a single state's fish and game licensing. We had 4 product managers (backend, admin, retail-front-end, consumer-web), 6-8 architects, at least 10 project managers and probably 70+ engineers, 30+ QA engineers. It had its specs written by a separate group for 6 months, that was then thrown away, and then rebuilt using Rational Unified Process. It was 10x over budget money-wise and 3x time wise. It took around 3 years to release to the public.
Those four products,
back-end team wrote in pure Oracle stored procedures.
the retail-front-end team wrote in 95% JSPs, with a small library of java functions. When logic needed to split amoungst 4 types of licenses, they copied pasted an entire 2000 line JSP over and changed minor code, then just forwarded to the right JSP depending on the type.
The consumer-web code was written in Perl, which needed to call functions in the retail-front-end, so the solution was to wrap the JSP code, call them with, what we would call now, mocked request/response objects. The PERL software called this mess using SOAP. So care had to be strip down every interface to use primitive objects so PERL and Java could talk.
The admin-code, my team, wrote in our own Java MVC framework, Struts was just created. What started as a web-project turned into a 'recreate the mainframe interface using html' though.
So yeah, 30 years ago, we did things right, or perhaps you meant 50 years ago?
First, I don't get the primary complaint in the article. The MTA chose OS/2 in the early 1990s, when there no other choices and it has served the MTA for almost 30 years with minimal hassles of upgrades or outages. I can't recall a single day when the Metrocard was not working - sure there are problems but it is more a UI/UX issue - a app that shows you Metrocard usage and balance would be insanely useful but we lack small mercies but hey you can pay with your smartwatch /s.
A piece of technology that is stable for 30 years is remarkable. Even if they had gone with Linux/NT the upgrade cycles themselves would have wreaked havoc. OS/2 seems like a brilliant choice in that regard.
Questions left unaddressed:
- What tech did they move to now with contactless payments?
- The move to contactless payments is awesome but what happens to people who lack smartphones or do not have credit cards or <big-tech>-pay setup on their phones. Or just casual tourists visiting from overseas with no US cards. How do they pay? Do they keep supporting metrocards forever? Does that mean to support Metrocards they need OS/2 and assorted new systems.
- What is the mainframe part of the stack and what function does it do.
“The OMNY system is designed by Cubic Transportation Systems, using technology licensed from Transport for London's Oyster card.”
https://www.cubic.com/news-events/news/cubic-wins-contract-n...
I love Linux and use it everyday but it would have been a poor choice in 1990 and some would argue would still be a poor choice today given other industrial OS-es on the market.
https://www.theatlantic.com/technology/archive/2015/11/why-d...
"Why New York Subway Lines Are Missing Countdown Clocks"
and how it's actually very about how managing a software/IT project can go wrong.
They don't need to stay with equipment they delivered at first; they just need to keep what visibly works working. They can replace older equipment and software with what works better in a bigger system, as they grow. If that seems more expensive, look at the hundred-million dollar+ contracts. Electronics is cheap, under those numbers.
> During a swipe, the MetroCard is read, re-written to, then check-read to verify correct encoding... Each MetroCard stored value card is assigned a unique, permanent ten-digit serial number when it is manufactured. The value is stored magnetically on the card itself, while the card's transaction history is held centrally in the Automated Fare Collection (AFC) Database... Whenever the card is swiped at a turnstile, the value of the card is read, the new value is written, the customer is let through, and then the central database is updated with the new transaction as soon as possible. Cards are not validated in real time against the database when swiped to pay the fare.
There's also an old NYT article [2]:
> But since the main computer communicates regularly with both the turnstiles and the booth computers, false cards have so far all been invalidated after they have been used once... Some hackers believe that because the turnstiles, the token booth computers and the mainframe computer, respectively, upload and download information at six-minute intervals, the length of time that a passenger can transfer without paying may actually be 18 minutes more than the advertised two hours, if the timing is right... ''I believe that, fundamentally, a system as complicated as the Metrocard cannot be absolutely secure,'' he said. ''What we have to make sure is that the hack won't scale. If a few people ride for free, that's no big deal.''
[1] https://en.wikipedia.org/wiki/MetroCard#Technology
[2] https://www.nytimes.com/1997/08/11/nyregion/what-galls-a-hac...
There are only turnstiles that can be jumped at the entrances that are manned by an attendant 24/7.
Haven't seen it in a few weeks though. It's expensive to be posting two cops near every turnstile entrance.
On that note: the city actually spends more on the NYPD for fare evasion than they could possibly recover from it[1]. I'd rather the city just give every teenager a free card instead, and spend the savings on a tipline for complaints about NYPD officers on their phones[2].
[1]: https://nyc.streetsblog.org/2019/11/14/mta-will-spend-249m-o...
[2]: https://www.thecity.nyc/2022/7/11/23204575/new-york-answered...
They were chasing and catching literally everyone. Well, the cops at the turnstile would radio the cops on the platform if they ran.
The people caught were really pissed about it too. Lotta loud arguing every time I went into the subway since they were catching people constantly. Granted I live in a neighborhood where people jump the turnstiles a lot.
Yeah, no shit. Fraud is YOUR problem. Not mine.
Similar scenario: chicken in the grocery store that advertises itself as "antibiotic and hormone free". No chicken in the grocery store has added antibiotics or hormones; it's illegal.
A-F have to all apply.
Visa, Mastercard, Amex, and Discover all additionally enforce $0 credit card liability on the banks that use their networks.
You'll only see the $50 thing on debit cards, and every bank I've used has waived that to give credit card-style protection.
On debit cards, the statutory limit is actually significantly higher ($500) if you don't report the fraud within two business days, and unlimited after 60 days, but as you say, I don't know of any bank that actually makes use of that provision.
EMV (chip) bank cards can default to local authorization mode if the transaction value fits your profile if the pinpad can't communicate upstream.
Nothing worse than folks not able to buy/use your system because it is "offline".
I doubt they do this, but if the data's encrypted with a good cipher used correctly, all you'll be able to detect is that it's changed.
Can you elaborate on what the difference was between how the licensing worked on OS/2 vs NT 4? What was the big innovation?
https://www.itprotoday.com/windows-78/ibm-microsoft-warned-p...
Other articles report that computer producers were given Windows "for free" as long as they did not offer nor support OS/2 on their hardware. Microsoft applied this practice for a few years and indeed lost income by giving Windows for free, but it ensured that the regular user never heard of OS/2 and then when IBM gave up the situation changed. Besides, Microsoft would still make money from Office and other products.
That was major design damage, required by IBM because most of the PS/2 line was on this CPU.
Both Linux and NT required an 80386 with a flat memory model. On capable hardware, this was far superior.
Both Linux and NT quickly ran on multiple platforms besides x86. I think that OS/2 was ported to Power, but nothing else.
"OS/2 1.x targets the Intel 80286 processor and DOS fundamentally doesn't. IBM insisted on supporting the 80286 processor, with its 16-bit segmented memory mode, because of commitments made to customers who had purchased many 80286-based PS/2s as a result of IBM's promises surrounding OS/2. Until release 2.0 in April 1992, OS/2 ran in 16-bit protected mode and therefore could not benefit from the Intel 80386's much simpler 32-bit flat memory model and virtual 8086 mode features."
https://en.m.wikipedia.org/wiki/OS/2
The developer of busybox/toybox also had 80286 discussion.
NSFT was still involved at 2.0 time, though DOS+Windows was pretty obviously going to win, so to speak, and NT OS/2 was still hush-hush. NT OS/2 ran on ia32, mips, alpha, and i860 (wow, freaky to remember that!) at the time - the "new technology" was as much the racy idea of portable OS code as RISC and microkernels...
VMS has great strengths, but hard file locks on OS components require downtime for patching. Cutler didn't foresee the avalanche of updates, and the availability collapse of "patch Tuesday."
POSIX opportunistic locking was a great gift to his competition.
http://www.edm2.com/index.php/HPFS386.IFS
That's the only 386 code in OS/2 1.x that I know of.
- It wasn't from version 1.1; the original OS/2 1.0 was a 286 OS, because IBM insisted it ran on the 286, because IBM sold a lot of 286-based PS/2 computers and had promised that they would run OS/2.
I have written about this recently, and you commented on the discussion.
https://news.ycombinator.com/item?id=33019019
That post was based on a comment here:
https://news.ycombinator.com/item?id=33007995
- It wasn't "forced down". It was originally a 286 OS, by design.
- "Most of the PS/2 line" -- [[citation needed]]. I am not sure about "most" but certainly 286 PS/2 were popular, because they were cheaper.
It wasn't about volumes; it was about an IBM promise to PS/2 buyers.
OS/2 only made it worse for IBM by shipping DOS+Windows compatibility, so basically zero developers wrote OS/2-native applications. This gave Microsoft nothing but time to get their Windows stability and multitasking performance up to snuff with NT.
In hindsight I feel like the dismal PC situation in terms of constant crashes/need for reboots in pre protected-mode MS-DOS and combinatorial explosion of incompatibilities created by all the random ass hardware peripherals conditioned everyone to fear more change and diversity. If the software being shipped on the computers at the time couldn't make things work well, and these were allegedly the experts most qualified to do it right, what will things be like with some alternative software attempting to emulate the same stuff in a "compatible" way? No way, OS/2 was doomed. And I'm saying this as someone who ran OS/2 WARP for years before discovering an Infomagic 4-CD Linux set in my teens.
Practically zero developers wrote for OS/2 because IBM made it hard to do so. I remember something Jerry Pournelle wrote about his experience at a major trade show (COMDEX?) that IBM and Microsoft both attended, after Warp was out and I think when Windows 95 was still in beta.
At the IBM booth he told them he'd like to do some OS/2 development and asked how to get started. They gave him a form to fill out to apply for their developer program. They wanted details on what he wanted to develop, his business plan, and stuff like that. If that was all acceptable to IBM then they would let him to buy their expensive SDK.
Then he went to the Microsoft booth and said he'd like to develop for Windows 95. They handed him a CD-ROM with the SDK, tools, and documentation.
Later, after Windows 95 was released I remember going to a retail software store (Egghead) and in the developer software section there was nothing at all from IBM. Microsoft on the other had everything one needed to start development there. Watcom C/C++ was also there, which supported DOS, Windows, OS/2, Netware, and major DOS extenders, so at least you could get some OS/2 developer tools at retail.
And if you did go through all the hoops and develop something for OS/2, IBM didn't seem to care. Apple and Microsoft were always on the lookout for 3rd party programs that could promote their platforms and would run ad campaigns featuring them. They'd pay stores to put such software in better locations.
(That same not bothering to market applied to the OS itself. The reason when when you walked into a major software store you would be greeted with a front of store Windows 95 display while OS/2 was on a bottom shelf somewhere in the back of the store most of the lights were burned out and the remaining ones were flickering in a way that made you nauseous is that the stores sell shelf space. Microsoft paid for premium space. IBM did not).
They also did a terrible job of hardware support, particular video card support. I wanted to quad boot between Windows 95, Windows NT, OS/2 Warp, and Linux.
I had to go through something like half a dozen or more cards video before I found one that worked well enough with OS/2 to support a decent resolution and frame rate. I had expected Linux to be the one to give me the most trouble, but it wasn't too bad.
IBM should have had its own team of video driver developers writing drivers for all the common cards that didn't already have good OS/2 support from the card vendors.
One can say a lot about Microsoft and Windows, but they were always keen on building a platform which enabled others to build upon.
Microsoft on the other hand always tried to give developers tools and access to it with quite good documentation. Enabling people to build on top of their platform. Making sure that other people can built upon their platform. This brought Windows in a ton of embedded worlds except the phone, where they missed the reinvention by Apple. But from say ATMs to Trains Windows is still ubiquitous. And that's the way they built Azure. Want to offer your own Service? - You can integrate it into Azures Cloud Console. And at the same time they still foster developers, with GitHub, VisualStudioCode and TypeScript.
Microsoft was sucked into the false idea that people wanted desktops on their mobile devices, but hardware at the time couldn't offer that in a cost-effective way.
Apple was also cutthroat pragmatic in a way that I don't think was ever in Microsoft's DNA -- specifically: killing Flash support.
The latter being firmly in the "scale top-down" camp: big customers are the only customers that matter.
I've worked for both top-down and bottom-up marketed companies, and the latter always seem to have better products.
I think it's because (a) there are so few whale customers, which hyper-specializes your product to their needs instead of getting a representative sample of the actual market's needs, (b) you can do dumb shit like limit your documentation to only customers, (c) it gives you a convenient excuse to ignore customer input ("Oh, they're not a customer that matters"), & (d) it slows you down, because your customers' upgrade processes are glacially slow, which removes pressure for you to be fast.
The cool part was that you could boot OS/2 off of a single floppy disk if you didn't need the GUI. I had that and Forth/2 on a demo disk.
Slackware came out on CDs.
Also:
> The Linux veterans among the readers will remember that Slackware 1.0 came on 24 floppy disks, 13 of which were the essential A series, while the remaining 11 floppies contained XFree86 and graphical applications. What exciting features could one find in Slackware 1.0? The Linux kernel was at version 0.99pl11 alpha. It came with math emulation and normal hard drive support, TCP/IP, support for ext2fs, msdos and several other file systems, and it even supported a PS/2 style mouse. It was compiled with libc 4.4.1 and g++ 2.4.5. The graphical part of the distribution was based on XFree86 1.3 and the OpenLook Virtual Window Manager was the default desktop environment.
Microsoft had us meet in Redmond, gave us a tour, free books from their bookstore, brought in an Indian chef for lunch (our boss was Indian), gave a big splashy presentation, etc.
IBM met us in a dingy conference room in LA, with a whiteboard, don’t even remember any coffee or bagles. The two guys they sent came across like bad car salesmen, and we were a big IBM customer.
If the sales people earned the same for renewals as they did onboard a new customer you can bet your sweet behind they’d treat you real nice like.
They're so vigilant about bribery that when someone (a general, or someone important) goes to a contractor presentation it's *expected* that there's a little can they put fifty cents or a few bucks into to pay for their share of the bagels.
And if this can doesn't exist? Everyone gets very uncomfortable very quickly - literal jobs are at stake.
Other industries are just wild, man.
Even if the financial part is 100% bribery, there's a difference between "here's a check so you pick us instead of the competition" and spending the effort to work out what your company/decision makers care about and providing that.
It's like saying if I plan a date with someone at a concert I know they're into that I'm bribing them into a relationship. In a literal sense, yes, but having someone care enough to cater to you isn't exactly meaningless in terms of deciding who to pursue a longer-term relationship with, personally or in business.
If you take a personal bribe to tank your business, that's between you and your business partners - a civil matter unless your business partners are John Q public (Which is why the SEC exists). It is entirely legal to enrich yourself at the expense of a business you wholely own, with the obvious caveat that the courts will pierce the veil if you've committed crimes as that company.
Somewhat related, to get my current job I had to sign a bunch of conflict of interest forms because I jumped ships from one side to the other. So some controls exist for the behaviour you brought up!
I'm not saying this stuff never happens, either, I'm just remarking that the professional standards and expectations are very different in military contracting versus software.
So there's a lot of arbitrary requirement shifting between actual requirements and as-bid program requirements... which invites arbitrary requirements adjustment on behalf of the bidders, but that can be explained away as {insert semi-justified reason here}.
The main problem with it was that is was basically unusable on consumer hardware at the time. We got a new 80Mhz 486 with 4MB RAM around that time and it was just incredibly slow, while Win3.11 ran just fine.
With hardware advancing at an incredible pace back then, it might have had a better chance it they had delayed it by a year.
Even though it wasn't a technical job per se, I learned to create a little Lotus Notes database to help us keep track of the various companies and more importantly, got my own copies of the rare developer manuals which were only available to licensees of Notes. Between that small bit of knowledge and IBM on my resume (a very big deal even in the early 90s), I was able to bald-face my way into getting a programming job and starting my career.
There was a reason, but not a good reason.
OS/2 used a lot of Star Trek code names. IBM forgot to get permission from the Star Trek IP owners before using "Warp" in marketing:
https://arstechnica.com/information-technology/2019/11/half-...
When is the last exploit you saw published for OS/2, many of the old OS/2 machines direct dial into an old modem bank (at a bank.)
The thread window on these is tiny because of how far behind they are.
It's OS/2. Hardly forgotten.
A commenter asked for an example of someone, anyone, still using OS/2 in production. I remembered this story and offered it as an answer of someone using it as recently as 3Y ago.
So I posted it as a top-level story as well.
I guess HN re-upped it and it caught on.
But yes, I think that for millennial techies, this is something that came and went before they were in the industry, so for them, it is forgotten tech that they've probably never seen.