53 karma · joined May 25, 2025
I was wondering why there's factor of ~50 in the ICE index formula. It turns out that factor was selected when the changeover from local currencies to euro happened. The multiplier makes the old formulation agree with the new formulation on the change date. According to Claude, the old formulation had exponents on DEM, FRF, ITL, NLG, BEF that sum up to the current euro exponent. I couldn't find a multiplier for the old formulation (I'm guessing it's selected to make the level 100 at some start date). I guessed a value around 12.5.
I did my best to make a DXY with a long history by stitching together free FRED data here: https://github.com/pandas-user/dxy/blob/main/dxy.ipynb
It's really interesting to see how much of an effect the Plaza Accord had on DXY. One thing I didn't check but might be cool is the ratio between dollar devaluation and positive stock market return during the three 1 year periods starting with the end of Breton Woods, the Plaza Accord, and the 2024 tariff thing.
I suspect that locating the referenced comment would require a semantic search system that incorporates "fancy models with complex decision boundaries". A human applying simple heuristics could use that system to find the comment.
In the "Dictionary of Heuristic" chapter, Polya's "How to Solve it" says this: *The feeling that harmonious simple order cannot be deceitful guides the discover in both in mathematical and in other sciences, and is expressed by the Latin saying simplex sigillum veri (simplicity is the seal of truth).*
At first I did not explore the tree visualizations in your web app: I simply noticed an index of company trees. Using the tree visualizer (which shows founders' names and pictures), I immediately realized that these trees represent complex human stories involving thousands of years of individual people's hard work. Interpreted that way, the data deserve a degree of awed respect that I did not show in my original comment. Truly sorry for that.
If you're curious about the shape of the startup ecosystem the way I am, there are a few things you could try. (In what follows I'm assuming "full graph" means company-company links with timestamps, not stories about individual people). pyvis has a feature that allows you to build a static html file with an embedded interactive representation of a graph. The data is embedded in the file, so you might not be able to share that unless you dropped enough information to conform such sharing to your data license. IIRC the static file has limited query/filter functionality so it can be difficult to make large graphs manageable for visualization. If that happens you can try using a graph database with a query UI. I remember another HN submission last year that (IIRC) used neo4j as a backend and provided a web UI with this kind of query/visualize workflow. I believe they also shared Github repos with the front-end/back-end code.
Anyway, thank you for sharing your project and sorry for the shit comment.
My favorite research code tends to look like the mathematics it implements. And that's really hard to do well. You need to pick abstractions that are both efficient to compute and easy to modify as the underlying model changes. My favorite research code also does the reader a lot of favors (eg documents the shape of the data as it flows through the code, uses notation consistent with the writeup or standard conventions in the field).
Industry research code... I'm happy to see basic things. Version control (not a bunch of Jupyter notebooks). Code re-use (not copy+paste the same thing 20x). Separation of config and code (don't litter dozens of constants throughout thousands of lines of code). Functions < 1000 lines apiece. Meaningful variable names. Comments that link the theory to the code when the code has to be complicated.
Overall it's probably most helpful to find a researcher in your field whose code you like to read, and copy the best aspects of that style. And ask readers of your code for feedback. I really enjoy reading Karpathy's code (not my field), but that may be an exception because a lot of what I've read is intended to teach a more or less codified approach, rather than act as a testbed for iteration in a more fluid design space.
I don't think there's anything natural about the mandatory use of copyrighted CUSIP identifiers in regulatory reporting. When SEC publishes its quarterly list of 13F securities it includes a disclaimer that it does so "with permission" from the copyright holder. My city doesn't pay royalties or seek approvals when it records and processes car license plate numbers for parking enforcement. The copyright holder seems actively involved in rulemaking that has the potential to diminish the role CUSIPs play in mandatory regulatory reporting.
https://www.federalreserve.gov/apps/proposals/comments/FR-00...
If you're adding more LLM integration, a cool feature might be sending the results of allow_many="left" off to an LLM completions API that supports structured outputs. Eg imagine N_left=1e5 and N_right=1e5 but they are different datasets. You could use jellyjoin to identify the top ~5 candidates in right for each left, reducing candidate matches from 1e10 to 5e5. Then you ship the 5e5 off to an LLM for final scoring/matching.
Kind of joking: unless there are nonlinear effects near 300K? Fig 4 [1] seems to suggest that the thermal diffusivity of the mantle grows very fast as temperature declines past 300K... but the data stop at 200K.
Reason for initial comment: we could probably set up a spherical heat equation to guess how crust cooling would change heat conduction at the outer core. But I have absolutely no idea how to reason about changes in heat conduction affecting the convection dynamics that generate the field. I was silently hoping for one of the domain experts lurking this forum to see it and share wisdom. (But overall it was a silly question, I know).
[1] https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/200...
For what it's worth: I ended up going the quant/finance route (as a "regular guy" with no meaningful accomplishments). If I could start over I would try to do something involving data analysis and biology. I think RNA sequencing is on an exp(-a*t) cost curve, and it feels like this is a domain where data analysis could produce something of greater value than slightly more efficient asset prices.
Trading costs. A cap-weighted portfolio manages itself. Square root cap weighting is going to trade a lot, and that's expensive.
So let's say your hypothesis is "stock go up on insider buy". Event studies help you test that hypothesis and quantify how much up / when.
Cool metadata table, I'm curious about the ticker source (Form4, 10K, some SEC metadata publications?).
My comment about CUSIP linking was trying to illustrate a more general issue: it's difficult to use SEC data extractions to answer empirical questions if you don't have a good securities master to link against (reference data + market data).
Broadly speaking a securities master will have 2 kinds of data: reference data (identifiers and dates when they're valid) and market data (price / volume / corporate actions... all the stuff you need to accurately compute total returns). CRSP/Compustat (~$40k/year?) is the gold standard for daily frequency US equities. With a decent securities master you can do many interesting things. Realistic backtests for the kinds of "use an LLM to code a strategy" projects you see all over the place these days. Or (my interest) a "papers with code" style repository that helps people learn the field.
What you worry about with bad data is getting a high tstat on a plausible sounding result that later fails to replicate when you use clean data (or worse, try to trade it). Let's say your securities master drops companies 2 weeks before they're delisted... just holding the market is going to have serious alpha. Ditto if your fundamental data reflects restatements.
On the reference data front, the Compustat security table has (from_date, thru_date, cusip, ticker, cik, name, gics sector/industry, gvkey, iid) etc all lined up and ready to go. I don't think it's possible to generate this kind of time-series from cheap data vendors. I think it could be possible to do it using some of the techniques you described, and maybe others. Eg get (company-name, cik, ticker) time-series from Form4 or 10K. Then get (security-name, cusip) time-series from the 13F security lists SEC publishes quarterly (pdfs). Then merge on date/fuzzy-name. Then validate. To get GICS you'd need to do something like extract industry/sector names from a broad index ETF's quarterly holdings reports, whose format will change a lot over the years. Lots of tedious but valuable work. Also a lot of surface area to leverage LLMs. I dunno, at this point it may be feasible to use LLMs to extract all this info (annually) from 10Ks.
On the market data front, the vendors I've seen have random errors. They tend to be worst for dividends/corporate-actions. But I've seen BRK.A trade $300 trillion on a random Wednesday. Haven't noticed correlation across vendors, so I think this one might be easy to solve. Cheap fundamental data tends to have similar defects to cheap market data.
Sorry for the long rant, I've thought about this problem for a while but never seriously worked on it. One reason I haven't undertaken the effort: validation is difficult so it's hard to tell if you're actually making progress. You can do things like make sure S&P500 member returns aggregate to SPY returns to see if you're waaay off. But detailed validation is difficult without a source of ground truth.
A few years ago I wrote an SGML parser for the full SEC PDS specification (super tedious). But I have trouble leveraging my own efforts for independent research because I don't have a reliable securities master to link against. I can't take a historical CUSIP from 13F filings and associate it to a historical ticker/return. Or my returns are wrong because of data errors so I can't fit a factor model to run an event study using Form 4 data.
I think what's missing is a serious open source effort to integrate/cleanse the various cheapo data vendors into something reasonably approximating the quality you get out of a CRSP/Compustat.
If options & futures are more liquid than the underlying, someone will be tempted to nudge the underlying.
Bond ETFs and their options chains seem like another locale where this could happen.
>90% of the work to replicate academic research in quant finance is data collection and cleaning. Academics have WRDS, Bloomberg, and research assistants. Others need to find cheap vendors of acceptable quality. Write scripts to fetch and clean. Actual algorithms and analysis (the fun part) are a tiny fraction of the total effort.
Project objective is "good enough" data collection & cleaning for daily frequency equities, futures, and FX. In that order. Annual data budget target < USD 500. Cheaper is better.