Model beats Wall Street analysts in forecasting business financials | Hacker News Reader

Model beats Wall Street analysts in forecasting business financials

news.mit.edu

218 pointsrbanffy125 comments

Hi former Wall Street enterprise software analyst here. Financial models were, generally speaking, tightly managed by the companies being modeled. For each upcoming quarter and full year, management (typically investor relations, sometimes CFO) would hop on a call with my team and discuss where our estimates were vs. consensus. Sometime, "consensus" was an internal measure and not what was reported by e.g. Bloomberg or FactSet.

If estimates were particularly out of bounds from consensus, they would politely ask how we modeled their business, if we would like help modeling their company, if we had a particular reason for out of bounds estimates, etc. That was a firmly worded but polite way to describe that the estimates might need some review and adjustment.

Hi former bulge bracket technologist/trader here as well. That seems like an extremely terrible idea due to the ability to pollute independent thinking by the model makers. What do you think about that?

I think that's the point though. Every growth software company wanted to have a beat and raise quarter, so they would manage expectations down to be able to offer exactly that, often no matter the quality of the earnings. As a sell-side analyst, if you wanted access to value-add with opportunities with corporates where you could get paid (e.g. non-deal roadshows), or even potentially banking business, you would generally need to have a good relationship with the company. The more divisive analysts would generally restrict contentious calls to one or two names that would generate call flow.

A lot of sell-side research work that analysts are paid for also focuses on information outside of estimates such as brokering investor sentiment or offering more details on channel checks in addition to what was published.

There was the whole era of “the whisper number” where analysts would publish one estimate and then feed journalists and certain customers another number. I specifically remember an analyst on CNBC literally say that he expected earnings on a company to come in ahead of his own estimate. Honestly, why is that number not your estimate then?

Whisper numbers are still totally around, but as buy-side expectations in my experience. I'd often field calls around earnings from investors who were trying to understand what everyone else was expecting from earnings results. There would then be further debates about what numbers would be good enough, or what long-only investors were expecting vs. hedge funds.

I remember (and will try to find) this one instance a few years back where one of the large banks issued a price upgrade shortly before earnings. This brought up the consensus prior to the earnings announcement causing the company to miss/underperform consensus which CNBC and other public news outlets weren't afraid use as the headline grabber...

The company management was pretty pissed, but it also shows how simple the public can be in interpreting the earnings miss/beat.

I no longer trade during earnings because it’s effectively random for John Q Public. A company beats across all estimates (eps, total rev, next quarter guidance) but drops 6% because a random non-public metric (like same store comp sales, or new subscribers added) that not a single analyst talked about that quarter was missed.

This is pretty common on the street. Not only for companies but buysides trying to manage estimates up or down.

Were parties aware that this was close to or a de facto reg fd violation (https://www.sec.gov/rules/final/33-7881.htm) and/or how was this handled internally?

Incidents and accidents notwithstanding, those involved in calls with and modelling for analysts and rating agencies are well aware of these rules. At least in my firm there is proper compliance coaching and continual awareness programmes about closed periods and internal continuity on what is disclosed publicly and when. Schedules are transparent and analyst calls in audio or transcripts are made available. Industry professionals even read these of competing firms, especially when there's trouble or a lol to be had. There's a lot that can go wrong, intentionally or not, but this part is quite a managed process.

Some companies like Microsoft were highly rigorous in managing information disclosure, and also in maintaining a lengthy quiet period. I could often hear investor relations at MSFT in the backgrounds of calls flipping through their internal playbooks of what they can and cannot say. Some other companies weren't as structured. I can't speak for how corporates managed disclosures internally, but I managed my own risk by typically publishing right after e.g. a corporate access event to reduce the risk of selective disclosure of material information.

I saw a handful of Reg FD filing updates and stock halts when new information was accidentally put out there by management. That was always embarrassing.

Could you explain more? That is like a hundred pages of regulations and I don't know anything about this.

Short answer: reg fd requires companies to disclose information material to investors to all investors. The previous post is suggesting that the close work with the bank analysts is conveying material information without proper disclosure. My hunch is that the legions is compliance lawyers at both the banks and at the companies have deemed this to be within the bounds of the regulation, but we’ll see if the SEC/US Attorneys agree.

Being cynical, I'd assume the "help" they offer is more likely to be stock market manipulation than disclosure of material information.

That's incorrect for these style of calls. The help is more akin to marketing - the company is arguing that they are doing better than the analyst thinks, and the analyst's company is offering their help to explain that to them and to other analysts.

That's exactly what I meant. They are not necessarily disclosing real information, they are trying to increase the stock price.

That isn't what stock market manipulation[1] means.

  Market manipulation may involve techniques including:
    Spreading false or misleading information about a company;
    Engaging in a series of transactions to make a security appear more actively traded; and
    Rigging quotes, prices, or trades to make it look like there is more or less demand for a security than is the case.

Presenting real information in a way that makes it more clear to investors is absolutely not market manipulation, and no interested party would ever claim otherwise.

[1] https://www.investor.gov/additional-resources/general-resour...

FWIW, my hunch is lots of information gets shared that the SEC doesn’t know about.

As long as breaking regulation is on average profitable the market dictates that regulation will be broken.

So depending on the risk/reward it most certainly can happen.

It makes me smile that you would ask this.

Where they aware everyone got bailed out in 2008, with nearly zero consequences?

The regulation you are expecting does not exist, in practice. Yes they wrote it down in a book. No, it's not real. It's as though I'm in a world of finance believers and I'm a finance atheist.

It does not exist.

Citadel and Two Sigma both have had large teams working on this for years. They are making a lot of money doing it but barriers to entry are very high. You need to collect all the data in a way such that you can reconstruct how it looked at any point in time. The vendors can’t be trusted to not make retroactive corrections, so you have to collect it for years before it becomes useful. Doing that takes a lot of time and money.

One of the wildest things I found while working on Wall Street was that ad blockers are wholesale selling data to analysts and funds. For example, I reached out to Ghostery asking if they had data about the data analytics tech / ad tech vendors used by industry, website, etc. to estimate market share and they responded back with a sales person telling me I could buy that data from them for five figures per year and that a number of my peers and clients were doing the same.

Hi, I work on the Ghostery extension.

Ghostery used to belong to a company named Evidon, which had this business model of collecting data to sell it in some form (the feature was called "Ghostrank"). In 2017, Ghostery was acquired by Cliqz (which builds an independent and private search engine from Germany), and since then a few things happened: (1) the extension was open-sourced, (2) Ghostrank was completely removed and (3) Ghostery is now developing paid products as a business model. For example we launched Ghostery Insights[1] and Ghostery Midnight[2] recently, both of which are subscription-based.

[1]: https://www.ghostery.com/insights/ [2]: https://www.ghostery.com/midnight/

Thank you for the update! I contacted Ghostery back in 2017, though I don't recall when exactly, and at that time I received a response from a salesperson. I was working on a project to try and understand market share of digital marketing software across the web at the time.

Ghostery seems to have fallen completely out of favor for this behavior, replaced by the EFF's Privacy Badger, actual ad-blockers, and even some browser built in functionality.

It would be interesting to know which other privacy plugins that sell data. Current recommendations seem to be pretty unanimously favoring plugins which do not, but it is always a moving target.

Hi, I work on the Ghostery extension.

In addition to my answer regarding the business model of Ghostery above: https://news.ycombinator.com/item?id=21897809. I'd like to add a few things to address your points:

(1) Ghostery did not "fall completely out of favor", as far as I can see. It is true that in _some communities_ (in particular some sub-reddits), some people tend to recommend different addons instead (often not based on any technical arguments, though), but this is not a trend that can be generalized (more people continue to recommend Ghostery as a very solid privacy protection suit).

(2) EFF's Privacy Badger is not a replacement for Ghostery, for more information about why, we wrote about it in the past[1][2].

(3) Ghostery has an "actual ad-blocker" built-in, in fact, it is one of the most efficient out there as was shown in a study that we published this year[3]. The adblocker as well as benchmarks are open-source and anyone can run them locally to verify the claims. We also more recently wrote extensively about the internals of this adblocker[4].

References:

[1]: https://whotracks.me/blog/how_cliqz_antitracking_protects_us...

[2]: https://0x65.dev/blog/2019-12-19/blocking-tracking-without-b...

[3]: https://whotracks.me/blog/adblockers_performance_study.html

[4]: https://0x65.dev/blog/2019-12-20/not-all-adblockers-are-born...

“If it’s free, you’re the product” strikes again

>Citadel and Two Sigma both have had large teams working on this for years. They are making a lot of money doing it but barriers to entry are very high. You need to collect all the data in a way such that you can reconstruct how it looked at any point in time. The vendors can’t be trusted to not make retroactive corrections, so you have to collect it for years before it becomes useful. Doing that takes a lot of time and money.

As someone who works in this space I assure you the barriers to entry are not that high. If I were to sort the various lists of accounts alphabetically you'd see names like Citadel and Two Sigma flanked by hoards of tiny funds you've never heard of. Many of these funds you've never heard of are 5-10man businesses or teams within larger funds. Making money is much more of an algorithms problem than a resources problem.

Can also confirm this - the research team I used to work on was just three people. We worked directly with some 30 or so funds, successfully, including Citadel and Two Sigma.

Did you find the work to be intellectually satisfying and creative, or was it somewhat rote in the sense of having to spend lots of energy on data hygiene?

I found it to be very intellectually satisfying and creative! But there were parts to the job which were also very boring, including cleaning data. Cleaning data took up probably about as much time as all the fun analysis and exploration.

What software is used in the cleaning and analysis?

By hand one Excel cell at a time :) It's why they get paid well.

I never used Excel, except to present reports. Data warehousing and analysis all took place with SQL and an RDBMS.

This is fascinating. Have you been working for a bigger company back then or was it an independent gig?

no disagreement that there is a huge long tail of small successful quant groups out there.

However, specifically in the realm of sourcing and processing novel raw unstructured datasets and creating signals that other firms don't have, as far as I know the big firms (some others in addition to the 2 I mentioned) dominate this area, as its too expensive and time consuming for smaller groups to do.

Dealing with the corrections is just another product. When I worked with IBES data, which collects all these estimates, you needed to buy a package called something like "As Was" to recreate any arbitrary point in time estimate.

It’s overrated. The alpha gets stripped out of the data sets very fast. Datasets are useful for about 1 year before the whole industry catches on.

The two funds I mentioned can afford exclusivity deals. When they expire the rest of the street catches on, by then they are already on to new ones. At any given time they have multiple in the pipeline.

I worked on an exclusive dataset with citadel. It’s never as valuable, the last few years were a special case where most funds were lagging on the analytics. At this point it’s nearly democratized. Also, even if you know exactly how a company is doing that doesn’t mean the stock will follow the companies real value. The whole thing is way more complex. Two sigma also doesn’t make as much money as they claim, they’re more of a market maker.

I can also confirm this. I used to curate data and develop equities forecasts professionally for about 30 or so funds, including Citadel and Two Sigma. It’s getting harder to build a successful trading strategy based on “quantamental” analysis alone (“alternative data”) each year.

A lot of fundamental hedge funds turned to this in the early 2010s as awareness of big data became a thing, thinking they could close the performance gap with the quant funds. It didn’t work. The quant funds that purchase this data use it as only one dimension of analysis to confirm a hypothesis which has already been empirically tested across many other inputs.

I have a specific example I can talk about, because my old firm abandoned the data: I found a reliable method for predicting exactly how many Model X and Model S vehicles Tesla sold well before earnings each quarter of 2017, including complete configuration data for each vehicle. Even with that KPI in hand, I couldn’t successfully forecast where the stock would go after each earnings call.

> I found a reliable method for predicting exactly how many Model X and Model S vehicles Tesla sold well before earnings each quarter of 2017

Were they leaking VINs somehow?

I think everyone leaks VINs via the NHTSA's site.

https://www.nhtsa.gov/vehicle

How would that give configurations? Genuine question.

https://www.autocheck.com/vehiclehistory/vin-basics

Super cool. Thank you!

Incredible. What strategies are hedge funds using now vs quants?

of course it’s more complicated than that - the end product is more features that can go in to all kinds of different trading strategies. For example as you said they are a big market maker - if they can more accurately predict earnings surprises with this data then they can price options more accurately in their MM strategies. Used properly it’s a lift to everything they do.

Like virtually every other attempt at predicting future events in complex systems, equity analysts -- even very knowledgeable, hard-working, well-connected, experienced ones -- are mostly just guessing. They regularly get beaten by a random dartboard in the WSJ, just like the majority of mutual funds trail behind index funds. Not much different than ESPN's former coaches and players -- smart, successful, domain experts -- whose sports predictions are not much better than random chance.

You don't have to be much better than random chance to make mountains and mountains of money.

You can actually be worse than random chance and still make mountains of money.

The fact is, for most jobs like this, it is far more important to look & speak the part -- to be able to make other people think you really know what you're talking about -- than it is actually to be accurate. Most popular weather-people are attractive and telegenic. Very few ever are actually measured by accuracy of predictions. Similarly, equity analysts who can impress the client base (who are, mostly, non-financial professionals) rarely get called out for lack of accuracy. Not saying it never happens, but most of the time the analyst (with the help of their employer) takes all the credit (for hard work, knowledge, insight, etc) when they are right, and blame "unexpected circumstances" or "unforeseen events" when wrong.

You can't be worse than random. If your estimates on a binary question are consistently wrong, all you need to do is to do the inverse of what your estimates tell you, and now you're better than random.

Irrelevant to this profession and discussion.

Sure, if you are selling the strategy and not executing on it then its performance is actually irrelevant, all that matters is your ability to convince others of its value.

The whole point of equity analysts is to impress clients (which is why they are made public), not for the banks themselves to take positions based on the research. Hedge funds who actually trade, don't release research or buy/hold/sell recommendations.

The entire sell side business is filled with sycophant analysts being paid under the table for producing good reports. The entire training set is bullshit if all you have as your "correct" dataset are these corrupt fuckers.

Do you, in so many words, concur with _sword's comment further up?

No. He's describing one side of the equation, which is companies forcing you to produce a favourable report. That is almost common knowledge at this point in the industry and he's absolutely spot on.

What I'm including beyond that is stuff like your brokerage arm asking you to produce a favourable report so clients will buy more of the stock. That's just one example. There are blatant and flagrant floutations of rules and procedures in these markets.

One recent example is Goldman producing a Buy report on tesla during the same week that their investment banking arm was placing shares on the market for tesla. Ideally you'd want a complete blackout on any sell side reports if your firm is placing the fucking shares, but goldman just didn't give a fuck and got away with it for that quarter (far as I remember, I didn't follow up beyond the next quarter).

I disagree in part with this. I had complete freedom to publish a report in contradiction with the firm's interest, which is also backed up by regulation. Investment banking and equity research on the sell-side is firmly divided by compliance and I could never ever speak with my colleagues in investment banking without the explicit supervision of my compliance department. That said, unless I was extremely highly convicted in an idea, it was usually better not to burn any bridges long term with the bankers or the companies.

Edit add-on: You're restricted on what you can publish while your firm is doing banking business with the covered company. The restrictions are stipulated by a mix of regulation and compliance departments. Typically during an offering you either can't publish at all, or you can only discuss points factually without changing your opinion or estimates. That however means that if you publish a report and you're buy rated, even if you're publishing a factual update on a stock you're effectively reiterating your rating. I'm not sure of Goldman's specifics, but they know what they need to do to meet research compliance requirements.

> but they know what they need to do to meet research compliance requirements.

Yes sir. And they were caught wrong footed. But I mean, the number of these "misdemeanours" they get away with has always been pretty high so it isn't surprising. They almost got away with 1MDB for fuck's sakes.

I don't know where you worked, but I was buy side, so I had full freedom to publish my own recommendations. However, my PM was a smart man and always wanted me to get on a call with the sell side analyst if I was disagreeing with him. But since I was very junior, a senior analyst used to lead the call (and was already tuned into the stock and following my models on the side because he was my mentor). Man the senior analyst (an ex sell side analyst from goldman) used to grill the ever loving shit out of the sell side guy on the call. Threw him charts from my report, charts from bloomberg / factset, news links etc on email while he had him on the call.

Like 3-4 emails exchanged just when we were on the phone. Sell side guy was also highly fucking experienced. Never fucking gave ground and admitted that he was being forced by the brokerage to mark it a buy. Always a stalemate and it was wondrous to me when I was new.

Then I understood they both knew it was going to be a stalemate long before the call starts. My senior analyst is cursing the sell side guy to me, and the sell side guy is probably cursing the two of us to his colleagues. We both agreed to disagree, but he never could fully justify his position in my eyes.

We had total freedom to give any rating we could justify. We downgraded banking names under coverage, or gave unfavorable ratings to some of the names that bankers wanted us to add to coverage. These sometimes led to calls from banking (moderated and monitored by compliance) demanding to know why we did such a thing, or my director of research receiving emails from companies that thought they deserved more favorable ratings that would personally insult my team. We nevertheless made the right calls for our investors to the best of our abilities, and I'll always stand by that.

I've been on a bunch of aggressive calls and they always get my pits sweaty. I was on a few with the senior analyst I started working under that I really remember. We had a contentious sell call on a stock and one of their top 10 holders wanted to speak with us. We hopped on the conference call and they brought every one of their covering analysts, PM's, and even a trader onto the call just to try and talk us out of our rating. It was one of those calls where we knew neither side would find an agreement, but ultimately we both got a better understanding of where our disagreements lay, which helped better inform both sides about debates in the stock. The cursing and aggression from these clients made that call especially tough.

On a different note, I found the former sell-side turned buy-side analysts to be the toughest clients to work with. They know exactly how the research game is played, but several seemed to have an especially large chip on their shoulder and wanted to badger the sell-side guys since they had taken that beating for years.

> On a different note, I found the former sell-side turned buy-side analysts to be the toughest clients to work with. They know exactly how the research game is played, but several seemed to have an especially large chip on their shoulder and wanted to badger the sell-side guys since they had taken that beating for years.

Very, very accurate from what I've seen. Especially from the former aggressive sell side people like the senior analyst I worked with. They somehow get redirected to the most aggressive current sell side people as well (despite stock coverage being assigned however).

My coverage finally didn't have too many contentious names (I took up defensives in fucking 2014 thinking the bull market was ending), but the ones that did had very mellow sell side guys. Whereas the sell side guys handling big tech and consumer discretionary names were aggressive as fuck. For instance, the same senior analyst took like 2-3 hours reading up on my stock before a sell side call, but with the consumer discretionary guy, he used to read up from the night before because (a) the stocks were more complicated, and (b) the sell side guys for these stocks were always super charged dudes with a ton of info directly at their fingertips. You needed to adjust your entire demeanour to deal with them differently.

Sellside gets paid for corporate access. They’re basically glorified cheerleaders. Especially in current market.

What really bothers me about these articles is that the authors claim to beat but they only study 37 companies. What about the 38th? Or the 380th. I guarantee you I can cherry pick 37 companies where a linear regression will accurately predict earnings quarter over quarter... what does that prove?

A study never proves anything. A p-value only tells you how likely the results are by random chance. Presumably, the companies were chosen beforehand. It is unlikely to have happened by chance that it beats for 37 companies. That’s different then selecting the companies after the results.

At the end of the day, securities pricing is based on supply/demand for the security. Predicting precise sales numbers isn't that important, stock prices move based on a plethora of other factors. How many times have we seen a company out-perform expectations and then the stock plummet, or a company under-perform and rally? It's cool that they're able to do this but it's not that big of a deal...

Of course stocks moved based on supply/demand, but there are drivers and catalysts which bring the marginal buyer/seller out to play. So I don't think saying that stocks moved based on supply/demand is adding anything really. If you're suggesting that all equities price changes are meaningless, that is simply not the case. If you're not suggesting that, my apologies, but the way you phrased it reminds me of something I often hear which seems to be a popular misunderstanding (and embraced in much of academia).

> How many times have we seen a company out-perform expectations and then the stock plummet, or a company under-perform and rally?

Far, far, far less than a company's stock surging after a good quarterly, a great product launch or tanking after a bad earnings report. You are nitpicking.

Yes, outliers and "irrational" market reactions exist, but there is an obvious arbitrage opportunity in deriving accurate sales numbers or other indices before the market gets its hand on them.

It is uncommon for a company to outperform expectations and plummet but very common for it to simply not react or dip slightly. Thus making it hard to make any serious money if you can predict outperformance

However it is more common for a stock to tank if it misses expectations. If you can predict underperformance, you can make some serious bank.

It's not the model, it's credit card data.

VISA / Mastercard could have its own hedgefund making great amount of money if it didn't sell the data to other companies.

Sell the shovels!

Maybe you are right, but having a monopoly on the data makes competition smaller and extracting revenue easier.

Their current position (apparently) keep them safe from antitrust.

Small data, Kalman filter, belief...

They're using Bayesian statistic. Probably added some expert belief to their prior distribution (informative prior). The whole article made it sound spicier than it has to be. The majority of the ML algorithms out there have too many parameters so it require a lot of data to overcome certain weaknesses like selection bias in tree ensemble, etc... So with small data it seems like ML are turning toward Bayesian. Also article may be missusing the term inference.

Do you have any recommended material to study this topic?

they do not use priors or expert belief. they solve the system id problem directly - which in most problems is unresolvable

I'm just working through the math in the paper right now. To me, it looks like all of the formulas could be expressed in a probabilistic PL and MCMC could be used to estimate the probabilities of both latent state and parameters. Would there be some particular roadblocks to this approach?

What does solving system id mean?

estimating model parameters. normally people use heuristics because it’s an unreasonable problem for linear dynamical systems without additional assumptions. here, they’re able to produce analytical results and finite sample analysis

*unresolvable

The headline itself demonstrates ignorance as to what matters to an investor putting money on the table. 1. Sell-side analyst forecasts do not drive the investment decisions of the more sophisticated investors. The sell-side is just a big marketing machine and the value-add of the analysis is very low at an individual stock level. 2. The model ‘beats’ the consensus forecast on a limited sample of names. By definition, the consensus forecast is an averaging out which leads to dilution of any one analyst’s alpha - hence it is not an appropriate benchmark.

It is a naive approach and study, but typical of academics who unfortunately have little exposure to real-world investing/trading strategies.

The use of ‘alternative data’ is not new and is definitely leading to alpha generation for some firms, but as mentioned by others such data-driven strategies will usually have limited shelf-life.

99,999 other models failed.

Let's see how well this model performs for n > 30 or across diverse industries.

Lots of comments about alternative data like credit card data.

But:

1. There are many factors that influence the movement of a security. "Short-term reality" (important KPIs basically) is just one element in the vector that determines price.

2. Getting long term exclusive alternative data is very difficult. Datasets are rapidly commodified.

Great. Now fire all those business analysts and replace them with software programmers.

Don’t tell me what’s in your model, just show me what’s in your portfolio..

Reply on news.ycombinator.com