Collection of Jupyter notebooks for quantitative finance | Hacker News Reader

Collection of Jupyter notebooks for quantitative finance

github.com

690 pointsCantaro8660 comments

Another useful resource, albeit more beginner level - Quant Economy course. It has both Julia[1] and Python[2] flavors. Recently they even added a small introduction to the data science[3]. And all of this - open-source project at GitHub[4]!

[1] https://julia.quantecon.org/

[2] https://python.quantecon.org/

[3] https://datascience.quantecon.org/

[4] https://github.com/QuantEcon/

I just worked on a project to predict significant(greater than 3 std) changes in the market and my biggest pain point was the lack of open source resources for quantitative finance(QF). It makes sense given the industry, but then I remind myself QF brought us Pandas!

Thank you for making this publicly available.

It's interesting that open source lives off transparency and equality (e.g. with git, anyone can at least start to put a patch together), whereas many other vital parts of society like finance, business, politics only work, when you explicitly do not share, what you know.

Politics would work a lot better if everything was out in the open.

It doesn't seem to make much of a difference, does it?

This is why computer science has evolved so fast, especially over the last 20 years, because of the open aspect of the code.

open source finance is crypto. arbitrage provides information everyone can use.

Practical example: get free price every minute from cryptomarketplot.com/api.json then you can figure out the EUR/USD spot using BTC/USD and BTC/EUR without ever needing yahoo finance or anything

if you're interested in pricing of securities, QuantLib is amazing.

Holy shit more gold! Thank you for posting this!

If you want to collaborate I am working on a lot of cool qfin related projects. Hit me at my email in the profile. Cheers

Thanks for sharing!

One thing to keep in mind before starting any finance project is the efficient market hypothesis. In order for you to make a profit relative to the market, you must by definition be better at modeling than everyone else. (There are certain caveats in terms of liquidity and leverage, but the general theme is correct).

I'll only note that beating everyone else is quite challenging, especially when you are competing against very well funded trading firms.

EDIT: Slight fix of wording as suggested by comment below.

the efficient market hypothesis is an axiom use to simplify models not an empirical fact. that's why it's a hypothesis not a law.

The joke is, two economists are walking down the street, and one sees a $100 bill on the ground, and the other says don't bother going to pick it up, if it was really $100, someone would have already picked it up.

A million pairs of economists walked down a million streets in a million possible worlds, but no jokes were made there because they didn't find squat.

The models here actually assume efficient markets and no arbitrage. You calibrate them to market, and if you use them to price a market traded derivative you hope they give you the market price you put in (if not then by definition they've gone wrong).

They are useful for pricing or replicating other derivatives without available prices, but not really aimed at getting an edge on the market.

Sure they are. If the price you calculated and the price in the market is significantly different, the law of no arbitrage is violated and therefore you have can fulfill your sworn duty as a market participant and arbitrage it away until everything is hunkydory again.

Thats also the reason why the market can never go to 100% passively managed money.

yes, but 99% is possible.

Possible, but would negatively affect liquidity and increase volatility. Trading would become more expensive as well.

What you’re saying is not what EMH says at all. EMH posits that there is only one Sharpe ratio: the market Sharpe. In other words, I cannot earn better risk adjusted returns than the market.

But anyways, no one actually believes that. EMH is used as a framework to price securities and as a way to reason about the market. Quant finance doesn’t work without the no arbitrage condition and therefore, EMH. EMH has absolutely nothing to do with “making a profit.” If fact, quant finance and EMH are built on a sure fire way to make a profit: the risk-free rate.

In short, while alpha is, by definition, a zero sum game, beta is not. So we can make profit pretty easily, and this is what most of the world does: obtain exposure to beta and make money.

Correct. I should have been explicit. By "profit", I meant profit relative to the market.

I do agree that the EMH doesn't exactly hold. Empirical proof of that is the existence of profitable trading firms. However, the reasoning behind EMH does imply something about the difficulty of earning profit relative to the market. And I do think it is a theory that people should be aware of when they start trying to understand finance.

Isn't the whole point of these forms arbitrage?

There is lots of data that shows that active managers don't do better than the market.

No, there is a lot of evidence showing that most active managers, investors and analysts can't outperform the market. There are counterexamples which demonstrate consistent outperformance, they're just the minority. Likewise most basketball players aren't good enough to join the NBA, and most players in the NBA aren't good enough to secure $10 - 100 million contracts.

The EMH doesn't even preclude the possibility of consistently beating the market (consistently mining alpha); it simply states that the cost of providing those investments as a service rationally rises to cannibalize the outsized returns, so it becomes a wash.

We see this in practice: the well known hedge funds which demonstrate consistent alpha eventually close their doors to outside investors. Why pool risk with external capital when you're printing money? Investors are a hassle and no strategy can scale infinitely. When you can consistently mine alpha it's strictly better to just become a prop shop and run on your own money.

How do you differetiate a manager that outperforms by luck from one that outperforms by skill. Data shows that managers that outperformed in the past are not more likely to outperform in the future.

You keep talking about this data, but you're not citing any of it. Therefore I'm not sure how to specifically counter what you've read.

But in the abstract, you differentiate them the same way you implement any hypothetical distinguisher in probability theory. Consider an n-sigma event observed to occur consistently. As n increases the likelihood of the event occurring by chance (rather than agency) decreases.

https://www.aei.org/carpe-diem/more-evidence-that-its-really...

There are so few managers that beat the market and then it might still be luck. Until you can be sure that a manager really is concistently better than the market he will be in his sixties.

We know factors that concistently outperform the market. So why not passively follow them?

There are so few managers that beat the market and then it might still be luck.

This is a naive way of doing the analysis, because we have examples of funds whose performance is so many standard deviations beyond the mean that we wouldn't expect them to arise by chance even if every single business in the United States was a professional trading firm. To get you started, I invite you to consider Renaissance Technologies, as one example. [1]

We'll assume that trading returns have a binary distribution. Traders win or lose with equal probability. This is not a great model, but it's good for making ballpark estimates, because it overestimates the odds of a track record like Renaissances.

RenTec's Medallion fund has not had a down year in the past 25. The odds of this are at most 1 in 33 million, using our binary model. Survivorship bias does not begin to explain this; there have not been anything resembling 33 million hedge funds over the course of history. I think 30000 hedge funds is a fairly generous estimate. [2]

In order to account for Renaissance's 30 year record of 70% returns before fees (and 40% after fees) under your hypothesis, we need to advance the claim that Renaissance has been successfully conducting massive fraud and financial conspiracy with a resulting profit north of over one hundred billion dollars over three decades. Even the common citation of the IRS case with the Deutsche basket options doesn't even begin to control for those kinds of returns; there would have to be something fundamentally novel criminal conspiracy occurring in Long Island.

Of course, you can still try to defend that position. But it makes the claim significantly more complex than simply saying, "most managers don't beat the market."

_______________

1. There are others. TGS, Baupost, etc.

2. This is copied from one of my favorite rebuttals of this point: https://news.ycombinator.com/item?id=9860254

I get that there is a very small window for funds that have alpha. However they are of no use to the average investor. Either they are closed to new investors or they are open and the performance edge goes away after some not too long period. Which is also the reason they stay closed.

Stocks don't have a 50% chance of being up per year. It is more close to 80-90%. The size of the return however makes a really good case that Simons fund is better than the other managers at discovering prices.

https://www.slickcharts.com/sp500/returns

I don't think we can be sure that Baupost's performance is not due to luck. Since 2001 it is reported to have an annual return of 9.4%, that is decent but not much above the market.

In the end, the average investor will not be able to invest in a fund that has consistent alpha. The alpha that a fund has will get smaller the bigger the fund gets. Just look at the recent performance of Berkshire Hathaway.

What's a factor?

Sounds like just another manager.

Factors are for example market, size or value. Over long periods of time they outperform the risk free rate or the market.

https://www.investopedia.com/terms/f/famaandfrenchthreefacto...

True for strong-form EMH but not for semi-strong or weak.

There are other factors than just beta. Size, value, profitablity and investements.

Are you aware of any in-depth explanations of the low-volatility anomaly? The best I’ve found is by Baker and they seem to point out alpha generation over long periods of time. I’m just a curious novice but I’m also wondering what it takes for an ‘anomaly’ to be considered more than a small aberration

I think this post is being unfairly downvoted. The idea is that making money via stock market trading is hard and that doing it consistently is even harder.

Sure, one of us might be the next George Soros, but it's important to keep expectations curbed.

> Sure, one of us might be the next George Soros, but it's important to keep expectations curbed.

He made his first big money by shorting the pound, which (to me, oddly) turned out to be a self-perpetuating burn-it-all-down machine. Which required that he had the resources to place the bet in the first place, but much like LTCM, kind of self-funded after the first dominoe fell.

So - great financial system hacker! For sure!

Stock picker? I'm not quite so certain. (It's possible you know enough more about this than I, that I could be overlooking some great "picking" behavior. I'd love to learn.)

I agree. But at the same time stock-market is not zero-sum because economy is growing, productivity is increasing due to innovation. So the question is how to take the maximum possible slice of the growth in the pie.

Can quants help? That I am not certain.

And many people like re-creational gambling. So why not gamble on a non-zero sum game, the stock-market?

It's only an efficient market because of arbitrage. If the price of copper futures in Europe suddenly soars, the price of copper futures in America doesn't magically, spontaneously rise to meet it; somebody has to do the work. You don't need to be better at modelling, you just need to be faster, or have a way of getting information earlier.

Not necessarily true; if you got a mark on copper futures in Europe then you would expect the best American copper futures bid/ask to change fast. Updating bid/ask doesn't require trading.

It requires active participants in the market.

Also useful is a Quant finance cheat sheet if you are going to sit an exam:

https://github.com/daleroberts/math-finance-cheat-sheet

Shameless plug, at Quantopian we have a bunch of materials (usually notebooks, sometimes also videos) covering introductory material in quant finance and Python: https://www.quantopian.com/lectures

(I work at Quantopian)

I'm always surprised at how people utilize Jupyter notebooks. What kinds of platforms/software do these firms use to perform the analysis?

It’s a bit of a mess right now. AWS is eating this market but frankly their products are not that great. Their ETL tool which does parallel execution and all is called Glue which is a cloud version of Spark. Glue is supposed to integrate with SageMaker which is basically your standard jupyter notebook experience. Spark though not that intuitive and is not the tool data scientists use for exploration. So data scientists explore and build model and then they rebuild them to run in Spark. Basically we would need a way to seamlessly scale pandas or R dataframes across clusters. Dask looks promising but it is facing an uphill battle vs aws and company and their inferior but convenient tooling.

A friend of mine is trying to build the databricks of dask for exactly that reason.

Proprietary software. This is mostly stuff used for options procing. Every options trading house has a substantial investment in proprietary options pricing software, mostly using customized adaptations of these models rather than what you would find in the literature.

Excellent, an area I am really interested in. Thank you for sharing.

This is an incredible piece of work, kudos for releasing it!

Thanks a lot!

Holy shit this is literally gold. I just went through this paper ("porting" to python)

https://epubs.siam.org/doi/abs/10.1137/s0036144500378302

I was planning on moving on to his book but now I'll just read this. Kudos to you Nicola for putting in the work and then releasing. If I were you I would typeset this and approach publishers because I bet you would sell many copies.

edit: i'll take this opportunity to poll the audience: does anyone know a good explanation of girsanov? i know a fair amount of measure theory but i'm still looking for a good, detailed, practical explanation of its use in the context of changing to risk-free measure for black-scholes.

cool paper, thanks

Thanks a lot!

Are you looking for a proof of Girsanov's theorem or an explanation of how it is used to price? A good reference is Oksendal [0], but it's still quite tough going. As for how it's used, a good reference is Shreve's second volume [1], which also contains a proof of Girsanov's theorem. Joshi's book [2] is a little bit lighter on the mathematical rigour.

A quick explanation of how it's used:

Taking the stochastic differential equation for geometric Brownian motion, apply Girsanov's theorem to change measure via a drift change such that we now have a discounted stock price that is a martingale. The discounted stock price is the stock price divided by a short term bond or cash account asset. In this new measure the discounted short term bond/cash account asset is also trivially a martingale since it's being divided by itself. So we have that our two key assets (discounted) are martingales. We then define the time zero price of the option (divided by the time zero price of the bond/cash asset) to be the discounted expected value of its value at maturity in this newly constructed measure. By construction this discounted option price is a martingale and we now have three assets that are all martingales which implies there is no arbitrage possible. With this option price, called the "risk neutral" price, no arbitrage is possible under our newly constructed measure, but, because the original measure is an equivalent measure no arbitrage is possible here in the "real world" either and so this is our actual price.

I appreciate there are a few steps here that seem like a bit of a leap. It took me a while to appreciate them. The key things to appreciate are:

How everything being a martingale implies a lack of arbitrage. Girsanov allows you to make your (discounted) underlying a martingale.

How you can then just make the option price a martingale by construction. And then how lack of arbitrage under one measure means lack of arbitrage in any equivalent measure.

The discounting can also be a little confusing, but it's really just incorporating the time value of money into the calculations.

[0] Oksendal B . Stochastic Differential Equations. [1] Shreve S E. Stochastic Calculus for Finance II. [2] Joshi M S. The Concepts and Practice of Mathematical Finance

Now THAT’s gold. Thanks for this nice explanation.

From my (a very layman) understanding, the price of the security is stochastic can be assumed only during a very short period, which does not include major moves. If this is correct, does it mean all these models are useful only for market makers earning money from the spread, or they can be useful for retail traders?

There are other models that take into account jumps in the prices/market. As for retail traders using these models, I think it is not recommended and not practical.

Any names/links/etc to learn more about such models?

The Heston model has stochastic volatility. You can add jumps to this: https://en.wikipedia.org/wiki/Stochastic_volatility_jump

I wrote up a simplified explanation of Girsanov's theorem once. You can get it as the limit of the ratio of two N dimensional Gaussian densities as N goes to infinity. I'll see if I can dig out what I wrote up.

i can't edit so i have to reply instead:

for anyone interested these two explanations i found when googling around again today are very good:

https://www.youtube.com/watch?v=omrnRshzHhw

https://www.youtube.com/watch?v=vKjbau2Hlrs

Reply on news.ycombinator.com