There is no more Perl at JP Morgan
thehftguy.com
thehftguy.com
Including many that seem pretty clear Perl is involved in a substantial way: https://jpmc.fa.oraclecloud.com/hcmUI/CandidateExperience/en...
https://db.wd3.myworkdayjobs.com/en-US/DBWebsite/job/New-Yor...
EDIT: since it was silently changed: This was originally titled and submitted as "Are banks still using Perl in 2020? The answer is no." (still visible in the URL slug)
(I won't comment on the Python 2 rewrite though)
I like Perl a lot. Out of the usual, interpreted dynamic languages, it is still my favourite. It's fun, incredibly expressive, got some things right early on (CPAN) and the community is probably the most entertaining out there.
It also caters to a kind of developer that no longer exists, or at least is in the minority now.
I will keep it as a sort of dirty pleasure for myself. Among the likes of Forth and Lisp.
The one developer I met who liked Perl was quite the character. He thought in a certain way I envied. Once he rewrote a bash script into Perl just for fun (and to confuse the rest of us). As a gratuitous side effect, the Perl script cut execution of the same task from over 1m in Bash to a matter of seconds.
I've resolved to learn it myself now, even if it nudges me more towards a mode of thinking. There's still plenty it can do.
I don't know whether any of them use Perl. I can tell you that if they use Perl, it has never caused a problem.
We use a fair amount of Perl, and it's a good tool.
I use perl daily. And Julia. And C. Python more like weekly.
Always select the right tool for the job. Changing code that works, and is critical should not be undertaken lightly. Most especially not of irrational dislike of the current implementation language.
Perlphobia must stop.
say("no");
# make it
die();
# Python's easier? That's a lie!I shall note that HN moderators edited the title 3 times making it more or less outrageous, nothing I can do about that.
Also it sounds like you didn’t read the piece as this comes right at the start:
> search covering repositories with 50M lines of code and thousands of servers
1. You notice how he said "Investment Bank". Guess what, that's not their entire business....not even in the slightest. Not even the most profitable. When I was there, they sold a significant portion of their investment services to buy more commercial, which is in an entirely different line of business. They aren't connected. Their systems are separate.
2. I worked there. I understand the absolute magnitude of how large their code bases are. The legacy software, the business processes. 1000s of servers, is nothing in a company that size. The division I worked for was a dust spec compared to the size of JPMC, and we had something like 300 servers on our own. We were a rounding error on the division's balance sheet. I should say, a division, of a division, of a division of a Line of Business. Multiply that by orders of magnitude and you begin to understand the size of their operations. Let me put this into perspective. A quick line count of the project I'm on, with 4 people and started 2 months ago (with npm packages granted) has over 500,000 lines of code. How big do you think JPMC's code bases are?
3. Do you honestly believe that a company that size tasked 1 person with just "removing PERL". Most of it was not being used anyway, and he "just did it." He just "searched" the repositories. How exactly do you think that worked? Do you think an EVP, a chief, just tasked 1 person with search "the codebases" across multiple lines of business? He might have done it for a division, a part of a LOB. I'll give him that, but not JPMorgan as a whole not even the the entirety of the investment arm. Think about how many legacy systems use PERL everything from legacy systems, to commercial ETLs. The amount of business knowledge held in those scripts, and making a mistake, even a minuscule one will cost potentially millions of dollars. What did he search? Did he do a Git search, or SVN, or CVS, or event maybe VSS? How many file and network shares? Did he have access to everything? All the devops stuff? What about all the things that are maintained by contractors and consultants? How about those?
What I can maybe say, is that the scope is touching JP Morgan, the corporate and investment bank with maybe 40 000 employees. It is much broader than you think.
"There's a joke among Perl developers that the next economic crash will be caused by a bug in someone's Perl script. Even then, all those redundant economists will have at least one employable skill."
There are like 3 to 6 major platforms used inside of JP Morgan, depending how you count (the oldest one is deprecated and being migrated away which will never complete, while the newest one is nowhere near working and will not have anything running for a long time).
The largest one regroups maybe 30% of all the developers and developments in the firm.
The Perl system was underlying on of these platforms, and has clients for use in all other platforms, so it was quite ingrained across the entire company really.
And ... did they actually, really do that, for a critical piece of infrastructure, because someone dislikes perl?
That should worry bank customers, shareholders, etc.
Business and tooling decisions should be rational, and risk reducing. I don't see that this change met either of these criteria.
I do think Serverless Perl could be a good idea though: https://metacpan.org/pod/AWS::Lambda
Why?
Also this is some weird post, is this blogger famous? Because it's not of much substance.
Servers will be upgraded to RHEL 7, so the application had to run on python 2.7 too in prevision of the upgrade very soon.
Then servers will be upgraded to RHEL 8 maybe 5-10 years later, so the application better work on python 3 too.
It is not glamorous but it's the practical way to make critical software work and continue working for some period of time. I think we can all agree that we don't want our banks to break every 3 months accidentally due to the newer release of whatever.
pyenv runs on RHEL 6 yeah? Does Python 3 not?
https://metacpan.org/pod/release/XSAWYERX/perl-5.32.0/pod/pe...
> There is no more Perl at JP Morgan, the largest investment bank in the US. The last Perl code was decommissioned in 2019.
I see the situation at JP Morgan now extrapolates to all banks globally.
I am glad that JP have done something sensible here, though.
> Nope, not python 3. Think 2018-2019 so no python 3 in sight.
What. That was literally last year. We all knew Python 2.7 was going to be EOL on January 1st 2020.
I know a lot of these legacy banks/financial institutions seem to operate in a time-shifted bubble a good few decades in the past, so maybe that explains it?
Yeah, it's funny though that the ones who've stuck to their aging mainframe systems and haven't invested in maintenance and reducing technical debt have the worst uptime.
https://github.com/openbankingspace/tpp-issues/issues?q=is%3...
https://twitter.com/nickrw/status/1141058572547215360
https://twitter.com/kieranmch/status/1188118150757224449
It's okay though, because they generally don't even tell the customers when it's broken.
I get status updates, good quality technical post-mortems and generally very good uptime from my modern bank.
So downtime for banks is very bad for business and for people's lives.
Note that this just means that the Python people won't support Python 2.7 going forward. I think you will be able to get Python 2.7 support if you pay for it for perhaps literally the next hundred years.
Decades maybe not necessarily, but years definitely. It boils down to the very real risk of being sued for damages if something's wrong with a payment.
I'm currently in a project which aims to re-write the front-end of a library of banking components - from AngularJS to Angular 9. The product is fairly simple to use, but still it's being delivered a year before AngularJS's end-of-life which will be after the 30th June 2021, because the time window for ensuring this thing works according to the banks' use cases must be as wide as possible.
Is this in the states? I get the impression the regulatory landscape is much more hostile to innovation and tends to favour incumbents (vs the UK, recently, where challengers have been supported a bit more).
The problem that I have seen in the enterprise/goverment world when I was in it, is them not keeping up from behind. E.g. They might be reasoanbly recent for a new project, but then dont update for a decade or so.
There is this recent introductory talk about enterprise frontend stacks on why it makes sense for them to stay quite far behind (compared to the pace in frontend dev) the latest developments that I found pretty interesting.
https://www.infoq.com/news/2020/06/second-generation-js-fram...
It's not hard to build RPMs of newer and/or supported versions of Python. You might even find it in EPEL. I get being risk averse, but if you're going to make the change, move to something sensible that your developers will want to work with.
Of course this still means having to switch again in a few years time but that's a problem for future JP Morgan.
There's "extended support", but no patches, not even security patches.
Edit: Apparently I was wrong about some security patches, but ELS sounds complicated: https://access.redhat.com/discussions/2399461
It's actually the hardest thing to do and to maintain long term. I'd say from experience that building your own is the #1 thing that was plaguing legacy systems in the bank and preventing to maintain them.
Developers built perl/python/whatever manually and distributed (RPM or whatever). It's de facto unmaintainable because it cannot be built again. Sure thing is, there were special build flags set and some extra libraries/plugins embedded into the build, but who knows? There is never any instruction left. Even if there was, these are complex software with huge web of dependencies, it's probably impossible to get everything to run/build 5-10 years later. (DLL hell is a major recurrent issue on Linux when upgrading the distro and some old libraries are gone due to googlecode/sourceforge dying).
Hence the solution to ensure that simple critical scripts work and can be maintained in the far future when required, is to rely on the /usr/bin/python from RHEL (and store everything in source control). Don't build your own interpreter.
None of the banks I worked for uses Perl to my knowledge, but that doesn't mean they don't use Perl at all, and certainly not that no other bank uses Perl.
But is there a problem if banks use Perl specifically? I'm no Perl fan, and I certainly wouldn't use Perl (or Python or NodeJS for that matter) to run systems handling money, but there's a lot more than just direct money handling going on at banks.
I doubt that.
For over 20 years people have been pinning their hopes on new languages and "paradigms" to make things better. Things haven't progressed a lot with that strategy.
I blame this on how software architects don't, and programmers can't, and because of that they all hedge their bets on new language X with feature Y, hoping it will make all the hard bits go away.
It's akin to how in the creative writing world, writers sometimes dive deep into the grammar and vocabulary of a particular natural language whereas the real challenge is thinking up new and entertaining situations and characters, and then filling that in with words in whatever language you happen to be proficient in.
Literate programming has been a concept for a long time now: write a document that explains, understandably, what the software is supposed to do, then accompany it with some source in whatever language the programmer is most comfortable with.
In a perfect world, that would be called "doing you job". In our current world, there is no money to be made in "doing your job" and it will be heckled as slow or outdated; praying for deliverance is much more profitable.
The too long, didn't read: Turing and Church proved that, fundamentally, all languages suck, so no new language will ever save one from one's own incompetence. Maybe it's time we all stopped blaming or praising "the language" wholesale for the various inadequacies and blind spots in our branch of industry.
- Are state unemployment agencies using Nim in 2020?
- Are aircraft carriers using Ruby on Rails in 2020?
- Are hospital bed firmwares using Excel VBA Macros in 2020?
There must be a lot of software written in <whatever language> that is still used inside the company.
At least in 2011 when I last worked in the industry, this sort of infrastructure was replacing the ad-hoc data-munging scripts that people used. It was even slowly eroding some of the Excel-based workflows designed by less software-engineery teams. 10 years later, I'm guessing it's only doing more than it was back then.
While client software could be python, java or C# (usage has evolved quite a bit over the decades), they ultimately called and depended upon a bunch of Perl on some servers.
Let's hope the windows 7 phase is over. I personally think perl 5 was a mistake, perl 4 was more fun and minimalistic ... just like javascript 5 vs typescript.
Commercial things have a way of ruining fun.