An oral history of Bank Python
calpaterson.com
calpaterson.com
Err....hi Sean, It's Armen. Uhh.... So I heard you like python.... well if someone was to uhh.... install python on the train... they probably wouldn't be fired. Ok bye.
Because it was unapproved technology I had a stream of people from technology coming to my desk to say I shouldn't have done it. I redirected them to Armen (who was a very important dude that everyone was frightened of).
The core engineers from GS went on to build the Athena system at JP and the Quartz platform at BAML.
//Edit for grammar
Seemed crazy to a new grad back then, but now I wouldn’t want to consider it either.
Thanks for your contribution! It was amazing that even in my role where I didn’t use Quartz, I could see and search all the code. Felt quite novel back then.
It took me more than half way down the article to identify I have not worked at the same bank as the author...
From what I understand, all this started with SecDB at Goldman, which was a the prototype for all these systems but wasn't Python based. The lore is that SecDB was instrumental in Goldman being able to rapidly figure out what their exposure was during the 2008 crisis.
Some of that team, lead by Kirat Singh went on to start Athena at JP Morgan and then Quartz. I met Kirat once, he was considered a bit of a rock star in the bank tech community. He now runs Beacon, which is basically Bank Python as a service.
https://www.crunchbase.com/funding_round/beacon-series-c--60...
We've also got a bunch of positions open too for those that are interested in joining!
Our CEO was also remote during that time too and here's him giving a webinar from a cabin :)
https://www.youtube.com/watch?v=fXPDXbrPdxI&ab_channel=Beaco....
Build complex system, when things starts to get messy move somewhere else. Rinse, repeat.
Second fun thing. Kirat was advocating for lisp for secdb for a long time and used to rag on me for liking python when it's so slow.
I do sometimes regret not taking the job because the people there were wickedly sharp and the tech sounded great, but in hindsight I'm not sure I would have thrived in a bank long term. I did a 3 month internship at Lehman's which I enjoyed, but I don't think I'd have suited a career in it. One thing I did get out of it was a total lack of fear around job interviews, if I could survive the 14 hours of interviews at GS and come out with an offer, then I can handle pretty much any recuitment process :)
1. An organization/industry can adopt each new technology only once. New technologies arise infrequently. Each time they arise, only a few people get to work on the projects introducing them. In other words, opportunities to leave a big mark are limited.
2. Credit for innovation is political capital. People hoard political capital and become powerful. They act as gatekeepers of innovation and take the credit for successful projects.
Correct. SecDB has its own language, S-lang, which you could probably best describe as a "lispy Smalltalk, with a hefty sprinkling of Frankenstein". The concept of SecDB was so good that other large banks wanted to get their own. Athena and Quartz have been mentioned several times in this thread, by people far more knowledgeable than I could ever be.
It's not just banking, I know of at least one large pension/insurance company who are building their own version of SecDB, with direct lines to GS. (They don't try to hide it, btw: the company is Rothesay Life.) The last time I talked with them, they were looking for Rust devs.
Disclosure: I work at Beacon.
Correct. We used python for a bunch of infrastructure stuff (eg distributing all of secdb to all of the places it needed to go). The actual pricing and risk was written in Slang with a lot of guis that were "written" in slang but actually that caused autogeneration of JIT bytecode that was executed by a JVM. Most of the heavy lifting behind the scenes was C++. So a bit of everything.
C++ changes went out in a 2 week release cycle so changes were still fast by most standards but much slower. But yeah we had 20m + lines of C++ code so it was extensively used.
Seeing Python at the core of trading, risk and post-trade processing for Commodities, FX, Credit etc was such a great developer experience.
By the time I left JPM, there were 4500 devs globally making 20k commits weekly into the Athena codebase. (I did a PyData presentation on this [1] for more details).
The one downside was the delayed transition from Py2.7 to 3; I left just as that was getting underway.
You need to be both meticulously risk averse, and also willing to do whatever needs to be done when it needs doing, and accept responsibility. It was great!
It was only a couple of weeks in, when I had to react within 60 seconds (USDJPY option expiry, NYC cut) on a position our front book would have lost MM on; with senior sales MDs screaming at me as well. Lo and behold, I was just used to it and could focus and execute based on my training.
My wife calls my thinking on the training - Stockholm syndrome. I still believe those skills were incredibly valuable for me, just perhaps delivered in a more 2021 acceptable approach.
Investment banks are basically risk-management shops. The partner made an assessment and evaluated the potential benefits as higher than risks. Note the word "probably".
Unlikely? Sure. But a lawyer somewhere thought it was worth reminding me 10x/day, so going to assume it's possible provided your unauthorized software caused a serious monetary loss.
Wow. Bell Labs too.
JSI (Java Slang Integration) was just getting off the ground but there wasn't too much for the front office tech teams to do there until it was to mature in the coming years.
Good times, thanks for sharing the history ;)
I worked a bit with Athena and was in the group that got Anaconda into the banking sector.
Small world we live in. I think I remember your name from JPMC days, but it has been awhile.
I could tell from the context it was nothing to do with QuickTime...
"Armen, did you tell Sean to install python?"
"No".
"Sean, did Armen tell you to install python?"
"Err... no. He said I probably wouldn't be fired."
"Well it turns out he's not right about everything. Here's a cardboard box for your things."
Armen must have known people would know Python had been put on these machines and that he authorised it, in fact what's the point of putting it on them if nobody knows and nobody uses it? I can guarantee you that within 24 hours someone was asking Armen why he'd authorised this and was justifying it. There cannot have been any possibility of dodging responsibility for this decision. If anyone got fired it would have been Armen, with a possibility of Sean going as collateral damage.
This is the big league. You make your decisions and you accept responsibility for them.
The OP said he told people openly that Armen told him he could do it when asked.
This makes no sense to me, what’s the upside to Armen? If he is business savvy, he needs to be gaining something in exchange for having his name thrown around by OP as signing off on this.
>In 2011 Goldman Sachs put its top computer wizard, Armen Avanessians, in charge of the division. He has helped turn round its fortunes. The arm’s assets under management reached a nadir of $38bn in 2012, but it now manages $91.8bn...
You say he reached out to you and asked if you liked Python. He probably wanted to roll out Python and was looking for someone who wanted to do it. If he told someone to do something they weren't passionate about, they would fail. He wanted to make sure it succeeded, so he reached out to you.
If he's such a bigshot and everyone was frightened of him, he must not have been afraid of them. When he said, "you wouldn't get fired," he probably meant what he said. He was giving you air cover. And it worked. When the gnomes came out after you, you just sent them to him. And they didn't bother you again.
I can imagine how the conversation went:
"Armen, did you tell Sean to install Python"
"No, did he"?
"Yes he did!"
"Great!!"
Now the gnomes are on their backfoot and have to defend why Sean shouldn't install Python If this guy Armen told you to do it, Armen has to defend himself to them.
One day I will sit down and write a small poem about the insanity of software development based on my experience with Quartz. It will be an intriguing story of love and hate being told through a sensual dance between sales and engineering. The battles will not be epic but the consequences of one’s actions will be far reaching.
It was indeed and experience worth having.
One such story, I humbly hope, would be my poem.
Yes the b00bs were all over the place as they are in real life, and in my opinion how they got there is as interesting as real life.
Original people solving the original problem had a good understanding of what they are dealing with.
However with subsequent generations it became a monkey problem https://m.youtube.com/watch?v=5QuwPeH9P7Y
When all you have is a DAG all solutions end up with sales person coming at your door problem. But years down the line we didn’t have traveling salesman problem but out of date libraries problem.
The monkeys tried to make their risk taking more safe by introducing all kinda of random constraints not realising that what gave power to the whole idea was actually risk taking.
And no, they didn’t try to manage risk but have completely fail to understand that risk is what made product good in the first place.
And this is how every great idea in human history fell apart, the original people had different view of a current problem however the following generations have only understood the simplified problem and down the line the the solution to original problem had became the actual problem.
I still believe the poem would make greater justice to the whole idea, but what I tried to explain here is that the simple things we have all witnessed actually hide much deeper truths about life in general.
And from there I assume QzDesktop loading times where not the issue at the beginning of this deterministic chaotic system, but simply one of the possible generational products.
I am yet to understand how to solve the monkey issue.
A favorite quip of his:
At GS, I'm like an arms dealer. When a desk has a problem, I send in the strats, and they blow away all the competition!
For example, you would write a pure function that defines:
Price(Security) = F(stock price, interest rate, ...)
This is loosely the same idea that drives React, ObservableHQ, Kafka, and other event-streaming architectures, but I first encountered this ~15 years ago at a bank.
I built a similarly reactive system for web UI binding back in 2004, running binding expressions on the back end with cached UI state to compute the minimal update to return to the front end, in the form of attributes to set on named elements.
To be honest, this whole paradigm seems absurdly fucking efficient for the developers. But I wonder about stuff like
* What happens if the data model needs to change? If you need to move something from db["some/path"]?
* How is it coordinated at a larger scale, how does everyone know what is running and how it interacts with everything else - can you figure out what depends on an object? What if the data used by your Price(Security) object changes and breaks it?
You write conversions and there's a registry where you register them to be picked up by the unpickler. If necessary you can also customize the logic that determines which version a given pickled datum uses to deserialize. There aren't so many guardrails when you're writing that stuff, but the infrastructure does its best to support you.
> If you need to move something from db["some/path"]?
There's support for both symlinks (db["some/path"] -> db["other/path"]) and for a kind of hardlink by making both paths point to the same inode-line id. You can usually find a way to do what you need to.
> How is it coordinated at a larger scale, how does everyone know what is running and how it interacts with everything else - can you figure out what depends on an object? What if the data used by your Price(Security) object changes and breaks it?
There's a common model for the things that are shared, and that has a versioning and release/deprecation cycle. Otherwise every type has an owner and you probably had to request their permissions to read their data, so you should have a channel of communication with them. But yeah people do rely on the fundamental business entities not changing too quickly, and things do break when changes are made.
See also the "observer pattern" [0]. It's a fun exercise to implement a reactive system in Python using the descriptor protocol [1]. IPython's traitlets library is an example of this in the wild [2].
[0]: https://en.wikipedia.org/wiki/Observer_pattern
Took many more years for us to understand that we had learned more about banking and making money than our masters had learned about the potential uses of the platforms we had built.
We started Sandbox Banking. Many of our friends are at hedge-funds :-(. What's the career paths of those that first built sec-db?
I remember Slang when I first saw the code, a parse tree based evaluator in 1997. Come on folks. Separate parsing from evaluation. Opaque types with message passing. Inference anyone? Clearly no one read hindly milner.
Add parse time optimizations, add locals, hey globals and locals weren't handled properly. Python in the 90s anyone?
Shitty KV store with 32K object limits related to some random OS2 btree limits. Add huge pages.
Deal with random rubbish related to inconsistent non transactional indices.
Figure out you should layer nodes in a dag. A dag is topologically 2 dimensions, fairly limiting.
Figure out that's somewhat similar to layering databases, it's just another dimension.
Hmm bottom up evaluators, you actually need top down as well, create a turing complete specification. well, limit it a bit.
Ah KLL points out that layers on top of dimensions end up being combinatorial, but you can actually cache the meta traversal and it's small n.
Lots of people point out category theory parallels. Haskell is pretty but completely unusable. I'm a math guy, and it's still unusable, I don't like feeling smart when I do simple things.
But interestingly creating imperative functions with pure input/outputs with implied function calls is pretty interesting. You can create an OOP paradigm with self and args as known roots aka linda tuple spaces.
Ah and each tuple space point can be scheduled independently, some issues with serialization and locality...
Go to another bank and choose to use python, foolishly decide to rely on pickle. Do that twice. Bad idea.
But write a much better petabyte scale multi-master optimistic locking database with 4 operations. Insert, Upsert, Update, delete. WAL as a first class API.
Finally decide that writing a coding scheme to convert python objects to json is not really hard. And of course cloud native and cloud agnostic is really the only way to go nowadays.
I'm always confused why people complain about Athena/Quartz, hell we wrote it all, fix it if you don't like it. Open source it if you want other people to contribute. If we made stupid decisions on pickling data, well there's a version id, add json serialization, it's not hard, don't take things as given.
This scared me a bit TBH. It’s one of those decisions that come back to bite us repeatedly.
Having a global view on everything is large banks' value-add, it's why they haven't been outcompeted by their more nimble competitors. Being able to calculate the risk of the whole bank isn't just a cool feature, it's the core value proposition of this platform.
Being able to just upload your code and run it is really cool, and if you squint it looks a bit like what the outside world is trying to set up with serverless/lambda-style platforms - just write a function, submit it, and there, it's running. (But it's worth remembering that Python is not a typical programming language; python build, dependency and deployment management is exceptionally awful in every respect, this isn't as big a pain point in other languages). Obviously there's a tension between this and having good version control, diffs, easy rollbacks etc. - but because Minevera is already designed to do all that for data (because you need that kind of functionality for modifications to your bonds or whatever), doing it this way strikes a much better compromise than something like editing PHP files directly on your live server.
What this article calls data-first design has a lot in common with functional programming. I hope that as the outside world adopts more functional programming and non-relational datastores, Minerva-style programming will get more popular. It really is a much better way to write code in many ways. The difficulty of integrating with outside libraries is a shame though.
I guess banks are the archetypal places that care only about feature creation and not about maintenance or technical debt. When something does break in the end, someone senior just shouts at the poor devs until it works again - usually fixed with a hasty patch again.
Similarly, documentation? Access control? Sanity? These seem to be left behind.
The biggest productivity gains were:
- having a single source of truth for both data and code (in a closely coupled environment)
- strong, battle-tested libraries to take care of all infrastructure concerns.
- enforced code dev/test/review/deployment workflows
This let the front-office devs be highly productive on adding real business value for their trading desks.
Remember also that these systems at GS, JPMorgan and BAML started around 2007-2010. The infra we all take for granted today at AWS/GCP/Azure simply did not exist back then, and banks' data security policies at the time did not allow cloud processing.
GS had „these systems“ well before 2000 (via J Aron). I think around the time you mentioned they spread to other firms (in their Python reincarnation).
It's not nirvana, these are real working systems built by humans with human failings. There are tradeoffs. Not every application is suited to these sorts of platforms, but the people building these things are top notch technologists and know what they're doing.
For example, changing scheduler jobs required submitting a change in Excel and having it approved (twice...) by someone. Except the table was world-writable and changes not logged. So in principle only your appropriate superior could approve change, in practice anyone could, and you’d never even know.
The thing is just because a team in a bank did this thing, that doesn't mean "The Bank" thinks that's a good idea. Like any company, banks are communities. I'm not making excuses, the fact this system wasn't properly architected is a failure of governance, but I've been on the other side of this trying to get teams to fix their problems and adopt resilient processes and procedures. Every offender thinks their service is special and their violation of the standards is justified.
Actually, it's probably not FAANG folks at all. I'd expect ex-FAANG folks to be more sympathetic to the forked python situation... FAANGs have an abundance of non-standard and frustrating infra (wasn't 5TB just posted yesterday?), and maybe even on steroids compared to banks (do any of the FAANGs not have at least one custom linux kernel?) Hell, both As roll a shitload of their own silicon.
Facebook literally maintains a python fork: https://github.com/facebookincubator/cinder
Google invented "NoSQL" before anyone else knew what it was, and all those "cloud" tools they used internally were obviously proprietary (except the ones they open sourced). Ex-Googlers I work with typically had to spend quite a bit of time re-adjusting to the "inferior" tools and processes in other companies.
Microsoft invented their own development ecosystem, and the only reason it's "common" or "standard" in the tech community is because they sell it as a product. This is the same for Apple at least for iOS development, and Amazon for their cloud service offerings.
When companies have millions of dollars to spend on maintaining a custom development environment that they think will give them a competitive edge, they will do it. It's the smaller shops that can't afford not to go with the flow, so to speak.
Is that how you'd describe the IDE and (integrated) VCS?
It depends which department you are in, but in general: absolutely not. Actually the reverse is true. Banks have huge risks to manage: just imagine for instance what damage a hack of their account system could cause. Or a crash of their payment system. Therefore it is of the utmost importance that systems are stable and bug free. In most departments, feature development is only in second place: stability and reliability have priority one.
This concern is so important that it is not just left to the responsibility of the banks themselves: for many systems, banks have to comply with external standards and are audited for that by external agencies.
I'm not sure how to react to that, but these features in Python are miles ahead of what many other languages have (or actually don't have).
It does if you put them all in the zip :)
(and build for exactly the platform your customer is going to deploy on)
In all the above languages, you simply ship a statically compiled binary (often just 1 file), and the user needs nothing else.
With any sufficiently complex Python project, the user will need:
1. virtualenv 2. possibly a C compiler 3. recent versions of Python (and that keeps changing. 3.0->3.4 are "ancient", and 3.6 seems to be the absolute minimum version these days --- due primarily to f-strings) 4. Or you ship a dockerfile and then the users need 600mb of Docker installed
I sometimes joke that in the future every Python script will require a K8s deployment and people will call it "easy".
Python is a great language, but deployment is a massive pain point for the language.
When I know I am writing something that has to "just work" on a wide range of systems that I don't necessarily control, well I don't write the solution in Python. I pick Go, Nim, or Rust (Zig would be a good choice too).
There are use cases where 100-300mb is no big deal and customer can handle this.
But single binary deployments with a statically compiled language where a fully-featured binary can weigh in from 5-30mb are what I'm after.
And honestly, with upx I can take even those fat Go binaries down from ~35mb to 7-8mb. That's an order of magnitude less than 100mb of Python and all it's dependencies. Not to mention with all those languages I mentioned (Go, Rust, Nim, Zig), I get multi-threading and high-performance as well.
I'm kind of surprised nobody monkey patched python serialisation to use a database (much like GitHub did with ssh key lookup in MySQL).
What does the devops there look like ? Snapshot every minute ?
For long-term persistent data ? Seems very dangerous to me, even reading a pickle from say PyPy vs a Cython intepreter corrupts the damn thing.
“Where is the code for that page?”
“It’s in the database”
“Oh… Like MySQL?”
“No. It’s an object database”
“???”
I called it “Martian Technology Syndrome”. But it worked. At later stages we paid the price and had to serialize the datastore for migrations, but that’s what you get for relying on pickles.
Joined JPMorgan in 2010 to work on Athena, and immediately had a real sense of deja vu... Athena's Hydra object db (essentially an append-only KV store of pickles) felt like a great grandchild of Zope's ZODB.
Specifically, until you realize that pickle changes based on python version, so updating from py3.x to py3.x+1 will prevent your application from reading previously stored data.
The Minerva I worked on was temporal and append-only, like a HBase that never did compactions (so "delete" actually just writes a tombstone row at a particular timestamp - there was an "obliterate" command but you needed special authorization to use that), and it was distributed (with availability zones even) so you didn't really worry about losing data; loading data as-of a particular timestamp was part of every query (and implemented efficiently). There were probably regular dumps somewhere too but I never needed to encounter those.
Interesting. Do you think you would do this today with a Cassandra/Hbase? Can it be done - let's say take python 3.10 and the latest Cassandra (or even better - something like Firebase or Cloud Spanner).
Just curious that in a post AWS/Firebase world, can something like Minerva be built, without investing in writing the db store ground up.
This assumption was challenged at the time by various groups - I was closest to the Equities Operations software team (although not part of it) who absolutely dug in their heels and refused to use Quartz. The assumption was explosively invalidated when people started implementing in Quartz applications that fell under Sarbanes/Oxley regulations and Quartz picked up a severity 1 audit finding - because Quartz was explicitly designed for "Hyper Agility" (literal quote from the quartz docs) - and anyone-can-change-anything-at-any-time does not make for applications that the regulators trust.
There was an interesting trajectory of Python hiring during my time at BAML. I joined just as Quartz was getting started and we managed to easily hire tens of python devs in London because it was easy to sell the fact that BAML was making a strategic investment in Python and therefore their (at the time relatively uncommon) skills would be highly valued. But as Quartz matured, Python developers generally came to dislike it (for reasons see original article) and it became hard to retain the best ones. And after a while Python 2.x became a massive embarrassment and, as Python became a more common skill in the marketplace, it became harder to hire good developers into BAML.
On the one hand the goal was to build a single enterprise scale system with a holistic view of the bank's data to do rapid ad-hoc position evaluations and meet new needs rapidly.
On the other hand, access to all that data and all the code is clearly a security concern. By the time I left the sev 1 finding was well on the way to being mitigated, but for example it meant that instead of handing out quartz developer accounts and IDE access like candy it had to be restricted to technology personnel only.
I reckon I "felt" that push to hire at one of the early PyConUK, where your boys suddenly showed up with a big contingent. I even thought about applying, but I was not based in London - and there were some red flags, like running a pretty old Python version (I thing it was 2.2 or 2.1, when 2.4/2.5 were the expected mainstream), that kinda sounded like I'd be signing up for the modern equivalent of mainframe maintenance.
Your post really made think it, it's an attractive area to work in.
If you're up for it, you'll spend a significant amount of time in the Quartz IDE. There are teams within App Support that develop monitoring and compliance reporting tools in Qz and do about 50% development. I know because I ran one. One of my team transferred into our dev team.
Depending on what kind of engineer, it is far better to go to the finance (front office quant, back office risk) side than the tech support side. They are less snobbish about autodidacts and pay is far better if you are willing to learn about things outside the dev sandbox.
(Our front office has a few quants and ex-quants with electric engineering background, I don't know of any software engineers there.)
(Front office deals with clients, and in this context comprises sales, trading, structuring. Middle office run control functions, reporting, risk, compliance, etc. Back office would be settlement, accounting, operations, etc.)
It was worse than that. What they actually built was a system designed to support complex hybrid structuring. It's what markets desks had been making a lot of money in prior to the crash esp GS. Unfortunately, post-crash there wasn't much money in structuring so the Front Office was more interested in investing in flow. Quartz was really, really bad at flow.
It took a long time (and the departure of Mike, Kirat et al) to get Quartz to a position where it was a reasonably sane FO system for the world as was rather than as it had been.
Fun times.
This is one of the most underrated topics in tech imho. Spreadsheet is probably the pinnacle of how tech could be easily approachable by non tech people, in the "bike for the mind" sense. We came a long way down hill from there when you need an specialist even to come up with a no-code solution to mundane problems.
Sure the tech ecosystem evolved and became a lot more complex from there but I'm afraid the concept of a non-tech person opening a blank file and creating something useful from scratch has been lost along the way.
But for 95%+ of analysis you really cant beat it.
Solution: the bank put together a system that constructs (from Excel templates and the bank trading data and market data) Excel spreadsheets from scratch every day, then used those for the calculations, and stored them. But now it was "a system", so all good.
* trader writes a pricing "app" in Excel
* trader discovers MS Access db
* traders (plural) start copying around Access db files
* problems
* "IT" gets involved
* convert MS Access to oracle or sybase
* write some server process(es) in C++
* write some replacement front end (spend months arguing over best grid component to replace excel) in C++/MFC
* trading system emerges...
* rewrite in C#, Java
* etc.
The mainframe mindset might be a factor here as well. The giant mainframe where all the magic happens is still a thing to behold and this is definitely part of banking's history and present. Mainframes are beasts and are still far from any kind of obsolescence. A monolithic Bank Python with a standardised set of libraries etc would slot right in to that mindset and way of thinking.
The part about programming languages frequently not having tables is interesting. The closest as mentioned is the hash, but you lose so much in that abstraction eg the relational aspects. The counter argument then becomes the obvious: why aren't you using a database library, or in a pinch, sqlite? Rightly so. Why would you add relational tables to python rather than have a generic python database spec or a collection of database connector libraries. Databases are separate and large projects in themselves.
I'd still be overly disturbed if they were running some old python 2.5 or similar. Just saying. That would be a source of pity.
The separate datastore is the problem to be solved here - databases, especially relational databases, are extremely poorly integrated into programming languages and this makes it really painful to develop anything that uses them. You can just about use them as a place to dump serialized data to and from (not suitable for large systems because they're not properly distributed), but if you actually want to operate on data you need it to be in memory where you're running the code and you want it to be tightly integrated with your language and IDE and so on.
(It's not even the main benefit, but just as an example of that kind of integration, when you're querying large datasets Minerva works a bit like Hadoop in that it will ship your code to where the data is and run it there)
The Python results of that first conversion need to test against Excel, so it’ll have identical lookup tables.
Hence "Active Record" ORMs like Rails and Django being highly successful. They functionally embed the RDBMS into the language/app (almost literally if using SQlite), which is a huge boon for developer productivity...
...but also a significant footgun, because it means the database is now effectively owned by the Active Record ORM and its (SWE) team, and not by some app-agnostic data team.
Want to reuse that juicy clean data managed by Django? Write a REST API driven by the app; don't try to access the data directly over SQL, although it may be tempting.
Right, those are a step in the right direction, but still a lot more cumbersome than properly integrating your datastore with your application.
And yes, it made for some very powerful high-level tooling. I actually learned to code on FoxPro for DOS, and the efficiency with which you could crank out even fairly complicated line-of-business data-centric apps was amazing, and is not something I've seen in any tech stack since.
> the efficiency with which you could crank out even fairly complicated line-of-business data-centric apps was amazing, and is not something I've seen in any tech stack since.
Did you ever get to try Delphi? Those "line-of-business data-centric apps" is what it was all about.
And I'm not quite sure, but I think and hope Free Pascal / Lazarus is close to that in ease and power.
This is covered in the article, in the distinction between "code-first" and "data-first". Databases means that you leave the interaction with data to a third party, and the only thing you do is send commands and receive results. This is very different from having all the data in your program, and starting from that. I'm not sure if "code-first" is the right word from it. Perhaps another way to put it would be that when data is the most important thing, you don't want to encapsulate it in a "database object", you want it to be right here.
The global distributed store of pickled python objects using Event Sourcing was one of the most horrible and expensive database systems I've ever heard of. It runs on THOUSANDS of expensive servers with all data stored in-memory. To get the state of a single deal you had to open, decompress, deserialize, and merge hundreds if not thousands of instances. And 90% of the output was more often than not discarded.
The Python interpreter extensions reveal the ignorance of Python by the original developers. There was no good reason to fork CPython.
There were many small subsystems created and supported by lone rangers with impressive CVs and astronomical salaries. A JIT better than any other one out there (but with a lot of limitations). A meta-query system extremely elegant.
But this all was a sham. The actual daily crunch/analytics was run on more classic SQL/Columnar clusters. From the distributed object database hours-long running batch jobs loaded stuff on old school DBs. And those blew up frequently. Sometimes those blow ups cost many millions in delayed regulatory reports. The queries running on top of SQL were beyond stupid and the DB engine could not optimize for them. And of course, people blamed SQL and not the ridiculous architecture and the OOP dogma.
Don't work for old school banking, hedge funds, or anything like that. They are driven by tech cavemen and their primadonnas. Exceptions might be some HFT and fintech shops.
PyCharm was a move in the right direction, but the way it worked was absurd- it would run the internal IDE in the background and sync to the file system. Given the proprietary IDE took up more resources than PyCharm, you constantly had to shut down apps so the machine had enough memory.
IDEs should not be a requirement- they are tools… but you had no choice but to use it and their totally flawed code completion. Their measure of success was tantamount to having a Jupyter notebook- write code and get back results immediately.
But as said I'd really love to work on those projects. Most of the people don't get the prestige to own one's own baby in a big corporation. 99% of the job is maintaining a shit mountain of code and piling new shits on top of it. It really took a lot of luck to be able to make it happen.
I mean even for people who get to work for Jetbrain or Microsoft Visual Studio team, they don't get to create new IDEs, they are buried deeply in a shit mountain of code and JIRA issues.
Plus the pay and vacation is really good for the banks.
There was also a bespoke time-span database. You could store keys and values in there, but every data point had a start and end time. Then you could query what the values were between certain times, or run numbers (operational periods). We used it for example to store what configuration the detector was using when a certain dataset has been recorded.
(* I've been out for a couple of years so I don't know what they use now, but I imagine it hasn't changed much.)
It was renamed…
Sometimes I'm constrained to a specific starting letter, so I've had to stretch it at times, like when I needed an 'S-' word... ended up going with 'Sinatra' since Nancy Sinatra performed 'You Only Live Twice' for that movie
Our PERT charts are really pert!
We burn story points not bras round here !
Agility is a core value ... nudge nudge ;-)
... oh well, The 1950s want their jokes back.
[1] https://en.wikipedia.org/wiki/K_(programming_language)
[2] https://en.wikipedia.org/wiki/Q_(programming_language_from_K...
[3] https://github.com/KxSystems/kdb/blob/master/holiday.q
[4] https://github.com/KxSystems/kdb/blob/master/c/c/odbc.c
[5] https://github.com/KxSystems/kdb/blob/master/c/jdbc.java
I am feeling the urge to learn array processing languages.
There is something very tempting.
They all use q and q is very wordy and highly readable if you speak english. It's mostly just developer defined functions which are compositions of the keywords of which there are not many: https://code.kx.com/q/ref/#keywords
Most of the code you would see in a kdb+ system in an investment bank won't look like any of the links you've provided.
Dependency graphs are an elegant solution to risk management and pricing etc. There’s a reason this approach works in IBanks.
Check out Beacon.io which is the a SaaS implementation from the same team.
Dependency graphs are not a solution to risk and pricing. They are, in certain circumstances, a very useful tool. That's all. They also scale notoriously painfully.
Putting a dependency graph as a mandatory component in your risk system was one of the worst technical decisions I've come across (and I've been doing this lark a long time).
Better as a solution to what problem? In some cases a dependency graph is an excellent solution. In some cases it's not. In some cases it's fine for small graphs but scales poorly as it can be very hard to reason about (as attested by pretty much anyone who's supported a really big spreadsheet).
But that's the point; it's a really useful tool. Sometimes.
The reason this was a problem was that it meant investment was needed for each of these things, and as such fell behind.
The IDE fell behind most modern IDEs, presumably because it didn't get the budget for it. The source control/ libs where usually modified versions of existing libs, but now needing to be maintained internally to remain compatible with the mainstream versions (which, again, they did not, and so fell out of compatibility).
> any transferable skills
It puts you in a position of arguing that you are familiar with <some-lib>, just a modified proprietary version of it.. Makes those conversations a bit more difficult..
> All were good devs but only knew quartz
To be fair - this is their own fault. It's difficult providing proof in terms of "what you worked on in your last job"; but there's no reason a professional python dev couldn't become familiar with the popular versions of things on their own, given they are fairly close in functionality. Many of the quartz devs I knew already had backgrounds in Python, attended pycons etc; so knew more than Quartz.
I'd expect with so many devs working on this the DevX will be ironed out
Consider a Python API that is a thin wrapper on COM calls intended to be used from Excel. Want to request some data? Fill in a 2D virtual Excel table. Want to pull some data? Query it and parse a text-dump of a table excerpt (remembering to parse #NA! Etc as nans). Want to automate a job? Enter it as a new row to a global spreadsheet. And for Gods sake, do NOT edit any of the other rows, lest the whole house go down in flames!!!
edit: /s we love you american colleagues,
We eventually did rewrite the Excel models in Java, released something, and then the whole project probably got cancelled or something, 9/11 happened a few years later and the whole building in which all this code was written had to be demolished.
This basically came about after the Nth time they asked for regexp help and I had a trivial solution that didn't work in whatever native implementation they were using and I basically gave them a choice between JendaRex and "not having me debugging their regexps anymore".
They unanimously chose JendaRex and everybody ended up happier as a result.
Edit: also applies to any large corporation.
My brief experience with this (in an adjacent area - proprietary trading) was that the more charitable view is that these firms need to be able to fully own their software stacks, and have the resources to pay for that luxury.
Reading these descriptions from the article, I can't help drawing a connection to the Smalltalk ecosystem. It sounds like, to at least some extent, what these banks have built is a system that exhibits many of the more interesting characteristics of an enterprise Smalltalk system, only on top of a tech stack that they could own from top to bottom.
Edit: For the record, I implement third-party vendor Excel functions (DLLs written in C++) in C# and it’s a great way to send useless processes to the shadow realm.
I work as a portfolio manager for a large reinsurance/insurance company but spend significant time in SQL, Python, Excel (not unexpected I'm sure).
The Wapole platform in the article struck a chord with me. We built something roughly similar - call it Trek - that handles jobs. Jobs encompass lots of tasks - reading/writing from Excel, executing SQL, running Python, running C#. I could list many limitations, but realistically, the biggest limitation is that the platform can't handle something that it can't configure and run. In other words, the platform isn't set up with R - so no creates data pipelines/jobs that use R. Lots of people here use R (among other tools)
One key problem (maybe?) is this all action happens inside the business. Trek was built by a talented actuary/programmer. No software engineering org involvement at all. I'm sure lots of folks here can imagine why: lots of red tape, general adversity to software that isn't already here, long stretches of time to get things done. Also, frankly, lots of our software devs write bad code.
For folks familiar with the orgs in this article, and other similar orgs, is what the article discusses happening mostly with software devs in IT functions? Are these folks embedded in the business? And also are there folks using the more technical bits of the systems that are business-oriented - analysts, investment professionals, etc.
Realize the lines are very blurry these days, but interested to learn from everyone here about the types/roles of end-users
I'd guess that you have proprietary (e.g. valuation / capital calculation) systems that need to interface with Trek in some way. Could you share how you've approached that at all?
Also not clear why R couldn't be added to Trek alongside Python and C#?
- We regularly perform group/segment level risk roll-ups. Involves running computationally expensive (by insurance standards) in-house and third-party models that estimate loss from hurricanes, earthquakes, etc. A lot of our insurance data is unsurprisingly stored in disparate systems that don't talk to each other, and in some cases, don't have any useful interface that someone like me can query/view. Things are changing, but still quite a lot of history to overcome
- We also have lots of stuff from outside underwriting parties in form of Excel, CSV, MDF files.
- We have to bring all that together to make sense of the portfolio, so we use Trek to do a lot of the various involved tasks like running the models, processing CSV data, processing Excel data, attaching databases, creating dashboards in PowerBI (tangent: hate it)
- Sample pipeline: query portfolio data from one DB, read CSV file from another third-party, pull both into risk model, kick off analysis, then execute a script to pull together results in the model's databases or elsewhere.
Happy to expand or answer other q's as you have them.
As for your other comment about R - it's just a matter of the install. Someone has to install R so that Trek can use it. Not a major problem. Pointed this out as a contrast in our org that is probably smaller and has many fewer devs compared to what I'm reading in the post where "bank python" sort of feels like the platform in which everything happens / everything is configured.
My background is mainly life which is dominated (at least in Europe) by computationally demanding proprietary liability modelling systems but I think Python / R is getting a foothold in capital calculation / aggregation.
My perception that there is a lot more use of in-house models in the GI / property-casualty worlds so more Python etc but sounds like you still have to interface with proprietary modelling systems.
There's not much (if any) appetite to completely rebuild 3rd-party geophysical vendor models. Those folks have 20+ years of work behind them and a different talent base (e.g. different types of scientists building the base models).
But we do focus on all the other stuff. Making data input easier/more accurate. Same thing re: output. Also the vast majority of our capital and group-level risk work happens in-house - R, Python, etc.
Have you considered developing a client facing API that can be used to send and digest data?
I don't know all the details but we do have connections to some of our MGAs through XML dumps or perhaps real-time feeds (I'm doubtful). But that data is often missing some of the details as I need. It's useful for policy admin - not for all the other stuff.
We've also explored portals but those are fraught with concerns about double-entry.
>lots of our software devs write bad code
Ultimately, you get what you pay for, all our full stack devs are making 6 figures... While it might not be FAANG money they also almost never work overtime and the stress levels are relatively low.
>Are these folks embedded in the business? No, thats my job. As a Tech PM I'm supposed to know exactly the business requirements, what my guys can do (to manage expectations) and any limitations of the software/business. I find the best PM's are the ones that have some Dev experience but also have extensive people skills and understand how to manage stakeholders.
>are there folks using the more technical bits of the systems that are business-oriented? It varies, I started off as that business-oriented person (corporate finance) and eventually made my way over to the Data side of things and finally some programming work and now I'm running projects. While you won't get many analysts/portfolio managers doing dev work I do try and get them to have a hands on approach especially when doing QA and UAT work.
Yes, it's the blessing and the curse. They are lovely people. They have lives. They aren't working 24/7. But there is misalignment between senior folks who want to innovate and build internal tech around core IP and the talent level of the folks tasked with actually getting that done. Insurance hardly unique in that regard, but an acute issue nonetheless. Firms like GS, JPM, etc have fatter margins (I think) and can afford to pay devs/strats/etc.
Interesting to hear that you went from corporate finance to technical PM. Quite the journey. Would love to dig more into that if you're willing.
From what I've seen on my end the Insurance firms have started paying somewhat of a premium to compensate for the lack of "excitement" that is associated with the insurance industry as a whole.
>Would love to dig more into that if you're willing.
Always willing to chat!
The problem with these projects is that the folks leading them have never built a real trading system in entire their lives (the ones who have been there for many years worked with end-of-day batch systems) and there is a layer of useless and incompetent "business analysts" who hide behind their incompetence by finding ways to malign developers..
Pro tip: Dont work for a bank before assessing its open source repos. They have none? Run in the opposite direction.
This rang a bell. How did deployment become such an arcane skill?
Our job as sold by the zeitgeist is to write code for features. Fix bugs. And run production.
The logistical part in the middle, how to get the code from commit to prod, is owned by noone and is not considered worth a budget.
There lie the reason we have tools to configure prod, but nearly no tool to deploy code. Even docker and k8s dodge that.
One of them is you don't own where your code runs anymore.
This might scare some people until they realize they can get high availability without waiting 3 months for some server to arrive, but it makes deployment harder.
I still believe the main reason people adopt CI/CD today is that "suddenly" deployment in a complex environment becomes easy and software gets tested. A lot.
That's not new. The first money I earnt from computers was making websites. We didn't own the webservers; we rented webspace from some company. To deploy it, we literally just uploaded PHP files to a place using an FTP client. It was that simple and it worked.
2. cloud providers try very hard to lock you in, by offering all sorts of advanced goodies. They tend to come with a learning curve, and they all accumulate. Sooner or later someone comes up with cross-provider solutions, and they too have learning curves.
3. inventing new ecosystems means creating new work for advocates and ninjas. You don't become a rockstar by diligently doing what has been done before, but by finding (or inventing) a niche and becoming a guru.
4. some problems are indeed hard to solve, and the more products try to do that, they more they get complex.
5. everyone thinks they will have Facebook-scale problems, even when they never will.
Prior to industry wide "solutions" such as Docker, or rather linux namespaces and cgroups, there was no obvious process isolation, so "deployment" was copying tarballs or using Windows installers.
Also, investment banks want "support" from vendors, so hardware was either Windows servers (mostly for Exchange and AD), or Sun Solaris boxes.
So although linux cgroups came around 2006 (?) and namespaces in 2001 (?), banks didn't do too much with linux until after 2005 (when Redhat were providing the aforementioned 'support'). I don't think the 'industry' widely recognised the potential of cgroups and namespaces.
I imagine something container-like with the notebook/batch-job (can be anything really), hooking up to datasources such as SMB-shares, thus allowing people who want to automate generating report Z to just request access to folder X for their job and thus be able to seamlessly create dashboards/... even if a lot of the org still is using "traditional" workflows.
> Financiers are able to learn Python, and while they may never be amazing at it they can contribute to a much higher level and even make their own changes and get them deployed.
Coming from a slightly different part of the finance world (insurance) this rang very true.
I think there is a huge opportunity here to build on the Python ecosystem - which is gaining more and more ground - and provide much more powerful alternatives to Excel and legacy proprietary systems.
context: https://www.techrepublic.com/article/jpmorgans-athena-has-35...
Model seems to originate from GS: https://news.ycombinator.com/item?id=29104401
The Quartz database is called Sandra (referred to as Barbara here).
The Quartz directed acyclic graph is called Dag (referred to as Dagger here)
The Quartz job runner is called Bob (referred to as Walpole here which is a reference to Robert Warpole whose shortname is..Bob)
These and the horrible proprietary IDE make it obvious which particular system he's describing.
Replication occurs automatically so you need to manage consistency in your app architecture. For example if you have instances of an app running in different geographies, the specific data for those instances should be in different folders.
"The sum of these values need to equal the sum of these values"
In that case you'd then needs to avoid
"sum_a == sum_b"
and use instead
"abs(sum_a - sum_b) < SOME_SMALL_VALUE"
Not to mention that there is a use case for auditing pricing models (as in external requirement, or internally) or comparing alternative models.
For a view of Python at JPMorgan, I did a talk at PyData London "Python at Massive Scale"
The python 3 migration is still ongoing
Thankfully, I think it can be said the light is now strongly visible at the end of the tunnel on Python 3 (but it’s not completely done).
My view now is the value of these in-house systems is essentially a cost efficiency play on run costs, and there there is very little revenue/growth opportunities for the business from these investments. With Volcker (really the best thing to happen in the '10s) and loss of prop trading means all the market makers live off the spread, and so while there is value to minimize operational costs, they are not worth the investments that have been made.
Bottom line for me is investments within large firms in capital markets are unlikely to generate revenue/profits in scale - I am sure there are some exceptions and would like to know
GemStone/S
"Mastering ENVY/Developer"
https://www.google.com/books/edition/Mastering_ENVY_Develope...
Some folk worked as "librarians" to promote code reuse across projects.
The one thing in there that really jumped out at me as "oh my god never ever do that" was using pickle for serious persisted data. I can see the upside around letting people who aren't dedicated programmers avoid thinking about serialization but...daaamn. My understanding is that this locks you into a "the API is all the code" situation where you can never change the data layout of any class once you've written it?
But - after a change - you have to choose either to leave that ‘backward compatibility’ in-place (essentially forever), or put together a job to run (on that scheduler! Hah) to go re-write in your new format. If you care enough, you might - and then you can remove your logic to handle the old names/shapes.
The charm in a lot of it is in its simplicity. It doesn’t claim to very smart, but people get it - and are often remarkably productive.
That sounds pretty reasonable. Not so different from tossing JSON objects into MongoDB or something like that.
Seems like a lot of the banking world like to invent their own tech stacks.
> Investment banks have a one-way approach to open source software: (some of) it can come in, but none of it can go out.
I wonder how well this plays with the various open source software licenses?
Likewise a bunch of fixes to the timezone handling code that iirc glibc simply wouldn't upstream so we had to maintain even though they were bugfixes.
We did used to upstream everything we could and I think the situation is improving.
Pretty much everything described is a Python library not a change in the Python interpreter so can be under a proprietary license.
The spirit of open source is a different matter.
I know some people think this is contrary to the spirit of open source, but it isn't. One of the goals of open source is so that users can customise the code to their specific use case, with no obligation to share. That's all the banks are doing. They have the same rights as any other user.
So it plays perfectly with the licenses. This is the sort of thing free software was designed for, allowing everyone that uses a codebase to own it 100%.
After Open Source, you still did that, but you also occasionally hired people to download and cobble stuff together to save money, and maybe one or two people to write code to help cobble together stuff.
Over time this evolved into managing entire "technology divisions" of people writing code and cobbling together stuff, to manage larger and larger internal projects, to support teams, to build components, to eventually be used by one internal product used by a customer. 50 different teams to build components, and 1 team actually servicing a customer. And each component built is exactly the same as components built at other companies. Sometimes even exactly the same as other components in the same company.
Nowadays a Big Bank might produce more software than Facebook, and none of it ever escapes into the world as Open Source. Whole oceans of software are birthed, live and die in the shadows. Millions of lines of code that live for half a decade. Constantly manufacturing their own hammers because they believe theirs will work better than an existing hammer, or because they're too lazy to learn how to use an existing hammer. And never sharing their custom-built hammers with the rest of the world. All because some clueless executives believe this solves their business case better than buying something off the shelf and making it work for their business case.
If you can prove & show evidence about this, I'm sure they will (grudgingly) accept the need to improve.
Financial institutions are mostly risk-averse, "if it works, don't fucking fiddle with it!" They need something that will impact the bottom line in an (almost) direct way.
I remember working with a financial institution back in 2010. I pushed for virtualization by presenting scenarios where the main servers are impacted, we have no warm backup servers, and calculate the impact to bottom line using known MTTR (Mean Time To Recovery) values of similar scenarios (gathered from incidents all over the world).
Took several back-and-forth meetings with the BoD, but in the end they accepted the need to improve and allocated the funding + greenlight the project.
(That was also back in the day when Microsoft had just released Hyper-V, and when we asked Microsoft to join in the project, I talked to their head engineer, and they respectfully declined because "their internal testing shows that Hyper-V, at the moment, is unable to fulfill the required parameters". Ended up with XenServer.)
I'd say thats how they see society and their role in it more generally. Doing God's work is a surprisingly directed graph. It applies also to the broader banking world, but investment banks being the most lucrative (when not bailed out) attract talented people that in principle can close the loop and return something important back (much more so than the "sleepy" commercial bank or credit union).
It would interesting if the above (potentially biased) view could be backed up by computing an open source leech ratio per industry sector. The amount of open source code used versus the amount contributed.
NB: a high leech ratio does not necessarily make you the worst offender. If your business model is evil no amount of open source contribution will wash it out
actually besides the tech industry itself I can only think of the bio/medical industry being an important contributor (E.g the entire R ecosystem)
Bulge brackets are more risk avoidant than smaller firms, like hedge funds. Today, we have LMAX disruptor and Real Logic’s Aeron (basis for Akka Remote) due to their liberal policy towards open source.
Then I read articles like this and I remember why I like it: it gets the job done, and quickly (for the developers at least). It's why it's so widely used and keeps climbing. Of course, nothing wrong with learning other languages and I do try to keep up, but Python will remain my go-to for the time being.
Same mistake as the people who keep talking about perl being "dead" while they're deploying their production platforms on debian or red hat based systems and ignoring the fact that the packaging and release QA work for those distros is substantially dependent on - actively maintained by the distros in question - perl projects.
Modern async/await + heavily OO based perl is not, I suspect, the language that you're thinking of when you made your comment.
Lots of "table-driven" systems live in bankland, and this python system sounds like the natural evolution of this...
'Barbara' (which I suspect is JP Morgan's Hydra) sounds like a brutalist version of the ZODB created with scalability in mind.
The hierarchical overlays sound like Zope's Acquisition and Containment.
Even storing (and running) code in the object database has it's equivalent in Zope's Python Script Objects.
I've heard this a lot, but have never really used Excel. What can it do that a programmer can't ?
Having seen Excel wizards work their magic before, the dizzying ways they can slice and dice their data with the help of a combination of GUI affordances, formulas, and hot keys is truly astounding. Often times a person could build out a full set of data and charts in half an hour that might be something like > 100 lines of equivalent Python/Pandas code.
And crucially often the report would have less bugs than the equivalent code because the analyst could see all the data in front of them as they were manipulating it and would naturally do spot checks along the way.
Now note the "some" in that statement. A Python/Pandas master could also probably whip up the equivalent > 100 lines in half an hour. But it was really astounding just how fast Excel experts worked.
But only ten lines of R! (Excel is kinda awesome though).
You've built this brilliant report because you're an excel wiz, but because of that you've gotten someone up the chain's attention and you need to do it every day/multiple times per day, and automating out all your shortcuts, hotkey and ui clicks with macros becomes a horrifying cludge that had you invested in something more automation oriented earlier would produce more resilient/repeatable solutions.
1. Tabular data. There's some tricks with named ranges etc, but for the most part your entire application state is spread out before you, scrollable in every direction. It's just tables, and clicking into a cell highlights relationships (data dependency) between cells.
2. Visible data, hidden code. =macros are hidden behind their result; the most obvious thing is to treat them as tiny black boxes, applying a single data transformation (or small set of logically related transforms), and immediately see the result applied to your data set. This is a tiny bit like functional programming, and a tiny bit like Pandas or Spark (immutable data, lazy evaluations). Except unlike those worlds, Excel pushes the data front and center, not the code.
And prob a bunch more I'm forgetting. It doesn't even feel like programming, more like building data pipelines in Unix or something. Except you can easily preview the data at each step in the pipeline. What I really want is Excel or Siag, but with Python and SQLite and a nice spreadsheet UI.
With code, you type the abstractions while imagining the data in your head -- and then check to see if the final result is the right one. The average user will usually litter the code with debug print statements to help understanding what's going on live
With Excel, you live in the runtime and you stare at the results of your "code" all of the time. The abstractions and the links between steps of the process either live in your head (because you know how that financial model works) or are buried in formulas in the most asynchronous of layouts (cell A1 may be the result of a calculation in cell Z99, for instance)
Wherever the author works, they employ a lot of Indians!