Hacker News Reader

Comment by cdavid | Hacker News Reader

I worked on that project 10 years ago as a consultant, and it was certainly strategic at that time. It was fairly well known that one of the main stakeholder was pushing for those system and making a career out of it by jumping bank every few years.

I understand the value of it, but as an experienced scientific developer and python dev, the culture shock was huge. I don't have fond memory working on it. It is very different from traditional programming environment, closer to a kind of reimplementation of smalltalk env w/ python. I believe an influence was actually an old system that used smalltalk in the 90ies at JPM.

One of the fancy thing was integrating reactive programming, which worked through ugly hacks, at least at that time, by parsing code to detect dependencies. IIRC, it could manage list comprehension but not loops. They also had their own python binary w/ both python2 and python3 in one process.

View on HN

The culture shock you mention was very real. I joined in 2010 (when Athena was 3 years old) and left 8 years later in 2018.

I remember my first few months unlearning normal Python and figuring out how to build the 'pixie graph', a lazily-evaluated Python DAG suited for calculating financial instruments. It took a while to get your head around this, but when you did it was a very powerful and productive way to building trading and risk management applications.

To get some sense in how this worked, here are two public projects on github with good introductions:

* https://github.com/timkpaine/tributary

* https://github.com/janushendersonassetallocation/loman

Sounds like a descendant of Goldman Sachs‘s SecDB/Slang with automatic dependency graph building. Did it have purple children (nodes whose value influences the graph structure) and twiddle scopes (modified copies of subtrees)? :-)

Just to add a bit of history, SecDB itself was an acquisition from J Aron, a commodity trading shop in the early 90s.

It was ubiquitous in GS by the mid-2000s, and was rather instrumental to GS navigating the crisis in 2008.

Being able to accurately and quickly compute risks across the entire firm's books, rather than manually merging across separate systems was key advantage, which encouraged JP, Citi and BofA to build a SecDB clone themselves.

https://www.goldmansachs.com/our-firm/history/moments/1993-s...

Yes indeed. Quite a few of Athena's early developers were from GS.

I've not heard the term "twiddle scopes" before; in Athena it is "tweaking".

Come to think of it, it might have been a “diddle scope” (I’m not a native speaker).

For finance applications (risk) the whole concept works quite well, doesn’t it - combining the natural advantages of code and Excel.

It was indeed "diddle," not "twiddle." At BAML, it's a "tweak." Beacon Platform is another implementation by the same team (but with support for the public cloud, and many more advanced features, I believe, including tighter web integration). I think it uses the terminology "bind."

[Disclosure: I was part of the Quartz Core team in 2011/2012.]

I was positively surprised that Slang was actually quite usable. I had expected much worse.

I suspect the Common Lisp influence was beneficial.

The weirdest thing, language-wise, that I noticed was scoping. It's neither dynamic nor static scoping, but weird scoping. (But there are work-around to get something like static scoping.)

Outside of the language, the whole CVS-based version control and review process was weird. But understandable as a product of the late 1990s, when review-before-going-into-permanent-history must have been way ahead of its time.

Slang was OK as a language. The SecDB/Slang ecosystem was years ahead of its time and all credit to its inventors and maintainers, but a monorepo full of decades of critical code from thousands of developers still gives me palpitations.

> The weirdest thing, language-wise, that I noticed was scoping.

Not the spaces in variable names?

Those annoyed me immensely. I can feel my heart rate go up 10 BPM just thinking about it. I guess it was meant to make scripts more readable to minimally-techy people, but actually just made it harder to parse.

No, actually not. I found that quite refreshing and occasionally useful.

But I was used to different and exotic conventions from the obscure languages I played with over the years.

Loman author here. Thank you very much for the mention. Amazed that I never heard of Athena or pixie graphs. Our intention with Loman was to create a library scoped for a single process - we looked at the possibility of creating a system responsible for executing much larger graphs on a real-time ongoing basis, but it felt like a larger project than we'd be able to execute well. It sounds like Athena was that, and it worked well, subject to being a culture shock for people coming into it?

A similar library from another asset manager - https://github.com/man-group/mdf. Although MDF seemed to work at the level of timeseries instead of scalar values.

I really like this video for explaining the DAG - https://www.youtube.com/watch?v=lTOP_shhVBQ

Man AHL also had a version that worked on top of nodes representing timeseries: https://github.com/man-group/mdf

Hi Steve, thanks for your talk and for the links. Are those repos pixie dependencies? Or are you saying that pixie works like those repos?

They are "toy" projects illustrative of the general concepts. ("toy" as in smaller in scope; one of the two is by a former colleague).

JPM's actual pixie code is proprietary, extremely performant after 12 years of pushing into bigger and bigger scale problems, and is definitely not on GitHub!

I believe an influence was actually an old system that used smalltalk in the 90ies at JPM.

That system you're thinking of was Kapital.

http://www.cincomsmalltalk.com/main/successes/financial-serv...

It's the only Smalltalk usage at JPM I believe, they could have doubled-down on it but instead went to Python and as you say, tried reinventing the wheel in it.

> tried reinventing the wheel in it.

That's not reinventing the wheel, that's building a new wheel you hope performs better/different with different materials and maybe some different techniques, and has a long and storied history of both successes and failures.

Reinventing the wheel would be if they published this system and some other bank recreated it without using what they published, thus "inventing" something that already existed and was available.

I never realised Kapital was written in smalltalk. Interesting

Apparently when it booted in the terminal, there was a nice ASCII portrait of Marx. I also have it on good authority that the program that launched the Smalltalk image from the terminal was called "das," so you literally would type "das kapital" to get the thing going.

> IIRC, it could manage list comprehension but not loops.

If it was like the predecessor system I worked with, it could handle either. But in a list comprehension, it could figure out the individual dependencies of the individual list items.

So, suppose you had a a function that took in a list, and returned a list with some computations performed on each element. If done with list comprehension, and one input element changed, only the corresponding output element would need to be recomputed, and only that would be recomputed.

If done with a loop, and one input element changed, the entire output array would be recomputed (even if all but one element remained unchanged - it would be hard to guarantee that property by code inspection at the time the dependency graph is built).

> python2 and python3 in one process

This would have had me running screaming from the building

I stared at my screen with deer-headlight eyes for half a second while hearing screaming in my head too.

AAAAaaaaaa

> I believe an influence was actually an old system that used smalltalk in the 90ies at JPM.

Imagine the benefit to the business had they just stuck with Smalltalk the whole time! It is remarkable how many businesses could have been quietly making money for years using a high-quality dynamic language such as Smalltalk or Lisp rather than going through the whole C/C++/Java/Python/Ruby/JavaScript/Go treadmill.

But I guess if they had done that then their programmers might not have gotten to type as many {}! Nor gotten to reinvent the wheel so often.

Kapital (the Smalltalk system) wasn't much of an influence on Athena. Athena was very much a SecDB/Slang derived effort.

And Kapital was an interesting experiment but rightfully died. Smalltalk just wasn't the right environment for something that complex and scale. I'd expect most lisps to suffer in the same way. And TBH it was more the image based model rather than the dynamic language per se though that was an issue.

There's a point where dynamism becomes a liability and Kapital went way beyond that point. Python fares a bit better but still, a huge effort is needed to control its failings at scale. This isn't helped at all by sticking a DAG in the middle of it as was done with Athena and its ilk. This all can be made to work but in the "cos its Turing complete" sense rather than the language helping in any meaningful way.

Source: I suffered through it.

> This all can be made to work but in the "cos its Turing complete" sense rather than the language helping in any meaningful way.

I had to LOL. You have very nicely put into words an experience I bet many have had.

> I'd expect most lisps to suffer in the same way.

Why? Eg Scheme or even the uglier Common Lisp are quite disciplined. And Clojure ain't too bad either.

As far as I can tell, Lispers don't usually go around monkey-patching things.

Elispers certainly do!

Oh, definitely. But no one in their right mind would use emacs lisp. Or any dynamically-by-default scoped Lisp.

Kapital was also rewritten in Java and survived in a few successor places ('Derivatives Studio' at CSFB). There it was my introduction to 'thunks'.

I don't anything about smalltalk, but one issue that was highlighted at that time was the inability to scale the number of "objects" to the required numbers (I can't remember exact typology, sorry). They used one of the smalltalk provider at that time, and went way beyond the upper limit.

That was the official story anyway. A lot of things were claimed when I worked there, a lot of it BS.

The culture in banking was definitely not for me: generally fairly smart people, but very perverse culture. This was the only environment I ever worked where people who actually lie to my face to make sure I made mistake when coding and fail. Everything was custom, and sometimes that meant athena, sometimes that meant terrible systems such as a "distributed database" that was a single threaded wrapper on top of an C++ STL hashmap running on NFS. The guy behind it was reworking its own protocol based on UDP instead of TCP because said system was too slow...

There was 0 abstraction, so if you wanted something such as EUR/USD pair, you had to ask the one guy who knew which id it was, so that you could get the data from in code. To this day I am convinced this was done on purpose for job security.

And all the time they saved they could have used to train new devs, reinvent a(nother) new vcs, figured out a new way to deploy image based software etc?

(I realize Smalltalk is great, but my point is it has its issues as well otherwise I recon sooner or later the advantages would be so clear that Amazon or Google or someone would be all over it since it would give them a competitive edge ...)

Reply on news.ycombinator.com