Comment by smallnamespace | Hacker News Reader

Comment by smallnamespace | Hacker News Reader

Parent Full thread

Armen was very passionate about the value of "strats" (GS's own term for "quants", and later broadened to include software engineers and data scientists).

A favorite quip of his:

  At GS, I'm like an arms dealer. When a desk has a problem, I send in the strats, and they blow away all the competition!

Also, SecDB's core idea is not just tight integration between the backend and development environment, but that all objects ("Security" in SecDB lingo) were functionally reactive.

For example, you would write a pure function that defines:

  Price(Security) = F(stock price, interest rate, ...)

When the market inputs changed, Price(Security) would automatically update (the framework handled the fiddly bits of caching intermediate values for you, so even an expensive Price function is not problematic).

This is loosely the same idea that drives React, ObservableHQ, Kafka, and other event-streaming architectures, but I first encountered this ~15 years ago at a bank.

It's as old as VisiCalc, it's how spreadsheets work.

I built a similarly reactive system for web UI binding back in 2004, running binding expressions on the back end with cached UI state to compute the minimal update to return to the front end, in the form of attributes to set on named elements.

Yes, although doing it in distributed fashion at the scale of SecDB or Athena introduces quite a few more complexities.

This immediately struck me when I was reading this article.

To be honest, this whole paradigm seems absurdly fucking efficient for the developers. But I wonder about stuff like

* What happens if the data model needs to change? If you need to move something from db["some/path"]?

* How is it coordinated at a larger scale, how does everyone know what is running and how it interacts with everything else - can you figure out what depends on an object? What if the data used by your Price(Security) object changes and breaks it?

> What happens if the data model needs to change?

You write conversions and there's a registry where you register them to be picked up by the unpickler. If necessary you can also customize the logic that determines which version a given pickled datum uses to deserialize. There aren't so many guardrails when you're writing that stuff, but the infrastructure does its best to support you.

> If you need to move something from db["some/path"]?

There's support for both symlinks (db["some/path"] -> db["other/path"]) and for a kind of hardlink by making both paths point to the same inode-line id. You can usually find a way to do what you need to.

> How is it coordinated at a larger scale, how does everyone know what is running and how it interacts with everything else - can you figure out what depends on an object? What if the data used by your Price(Security) object changes and breaks it?

There's a common model for the things that are shared, and that has a versioning and release/deprecation cycle. Otherwise every type has an owner and you probably had to request their permissions to read their data, so you should have a channel of communication with them. But yeah people do rely on the fundamental business entities not changing too quickly, and things do break when changes are made.

Thanks! Interesting stuff.

There's also a graph debugger that allows you to step through the dependency graph node-by-node across the various globally distributed databases.

True but not really helpful for this problem, because it can only tell you about the job you're debugging, whereas what you want to know is what code might ever depend on that data.

> This is loosely the same idea that drives React, ObservableHQ, Kafka, and other event-streaming architectures, but I first encountered this ~15 years ago at a bank.

See also the "observer pattern" [0]. It's a fun exercise to implement a reactive system in Python using the descriptor protocol [1]. IPython's traitlets library is an example of this in the wild [2].

[0]: https://en.wikipedia.org/wiki/Observer_pattern

[1]: https://docs.python.org/3/howto/descriptor.html

[2]: https://github.com/ipython/traitlets

Thanks for the links. eye opener for me, despite using python for serveral years.

Thank you so much for this context! When I started at Lehman in 2007 and for years through the i-banks & hedge-funds, 'Sec-db' was the North Star for so much me and my friends built. It's amazing to hear from the folk who brought that to life!

Took many more years for us to understand that we had learned more about banking and making money than our masters had learned about the potential uses of the platforms we had built.

We started Sandbox Banking. Many of our friends are at hedge-funds :-(. What's the career paths of those that first built sec-db?

Reply on news.ycombinator.com