nnechm

Had a similar experience with meta. Extremely opaque decision making, terrible UX. Account permanently disabled ... did not follow community standards, literally on sign up to get a dx account. It's difficult after an experience like that to see how they are so successful. Is it because their users are so addicted and ad sellers will do anything to get onboard ? It's probably just a few dark patterns here and there to bump up impressions at will.

Feels more and more like it's the best of times and the worst of times...

Things feel off,but you have (mostly well-off) people talking about how great things are going to be and are. I suspect a lot of this correlates to the stock market.

For the median millenial atleast, whatever you were taught growing up is just the wrong guide to understanding the world today.

I think social media by its very nature, has to be understood inverted. A high number of posts about how great things are suggests otherwise. Comparisons with Europe have gone exponential that tells you more about how the posters feel than anything else...

Thank you for the advice, will implement accordingly.

I am assuming this is on Cofin and not on beatandraise.com, if so, please let me know, I shall fix it. You have 20 chats etc.

:) I guess its not very odd to say investment research is a lot of reading and LLMs are already very good at reading. So there is little doubt that LLMs will change the investment research process. Have you used CapitalIQ, do you use it on the browser or in excel. If you use it on the browser, the experience using beatandraise.com is already better for some cases. For instance, try getting Apple's Rest of Asia Pacific revenues which apple has been highlighting is their main growth region this year, you can pull that out easily. https://imgur.com/a/rRnrB4u Can you do that on capitaliq, you would have to hope their standardized tables include this...

I don't think excel can be defeated :) I certainly do not expect financial professionals to switch to a chat interface.

When you login into capitaliq or factset or look at a bloomberg screen, you access data, you can then do the same here. Excel plugins go on top which can then get this data into excel to build models. The api that powers this app, can also send data into excel for instance.

Data can be copied already directly from the chat box into excel, maintaining the table format for e.g

Regarding standardization, I think data was standardized not to enable comparison but simply to fit into the same schema for every industry. Most companies within the same industry report the same way. REITs do not report like SaaS, but the existing datasets put it all into one set. Raw data from source is always better as you can convert it to standardized but you can't go back...

I think you are referring to the more hyped version where somehow LLMs can figure out how to get the most relevant information from 10Ks and do most of the investment process. My mental model is simply that we have an assistant, an amazing assistant that can read pretty well, and we can use them in the investment process.

Yes, information retrieval is hard :) A lot of people ask for 'can you get Apple's revenues by product category for all of 2020', how do you get the smallest piece of text that has the most information about product category and revenues ?I can get a lot of text, but that would mean I probably run over the context limit and so on :)

That's a good question, for e.g if you want to get Microsoft's revenues by product category or apple's revenues in greater China. https://imgur.com/a/LdQkt7j

You can also do this across time with constraints on context limits ...

What is a traditional method, would you search the string or would you find the named entity (using NLP) and look for the entity ? or do you mean a ctrl +F in the document ?

I guess the premise of LLMs, is that we have an intelligence that can read and write, so it can do this in an automated way in different applications. In this case, I am trying to automate and speed up the investment research process, along the way, we can create our own dataset of financial data that can be generated on the fly as necessary.

Also, how would you extract an income statement from the text, also in the same img using a traditional method ? I personally find it magical it can do that, it knows where it ends and so on...

A few things on prompting: 1. Get me google cloud revenues fails, because somehow gpt4 thinks i am talking about an entity called google cloud and not google :) 2. So in order to fix it, you can either ask for Get me google's cloud revenues or get me google cloud revenues from google's results...

As you can see, inspite of all the training in the real world, gpt4 thinks google cloud is more of an entity than google is, based on that question :)

Thanks for letting me know, i guess you use a different strategy of querying by company and by document. I see this when I try to get Apple's rest of asia pacific revenues for e.g

I guess it really depends on how your targeted user would use it... :)

https://imgur.com/a/uEgLUpz

Yes, they are definitely not primitive, but considering how the breakthrough in LLMs happened, i.e MSFT, GOOGL were working on it, even as late as 2021, Google's BERT was not really there (at least the one they showed the public in the blog from Google research). So the sudden jump from ok to great thanks to openAI would probably mean a lot of existing systems also need to do that.

Hi, Thanks for bringing that concern up. I shall keep it in mind and change it based on feedback from customers if it is an issue. Typically, people search for a company after a price move rather than before. And these are searches on publicly available data, ie data that is not proprietary and already filed with the sec. Needless to say, none of the chat traffic is used or will ever be used to feed any trading signals for anyone.

It would be possible for example to get AMD's revenues over time like this, it's tedious because of context size, but it's unrestricted, so you can get whatever datapoint you want...

If we run this query over an api for income statements for all 8k companies, we pretty much have all income, balance sheet, cashflow items, shares outstanding etc. Add stock price data, that can give you EV/Ebitdas, P/Es and all that stuff.

https://imgur.com/a/2U2wt4h

The functions tend to fail when the prompt is complex and the user asks for a lot of fields, and typically the last field in the json is not closed i.e missing a }, i guess openai is aware of it. It doesn't fail that often to have to write a workaround, atleast not yet.

So like a lot of applications, the problem boils down to being able to serve the right text. You have something that can read and do basic inference .... You need to tell it what to read so that it can answer your question. But it can only read 16k tokens (20k words at best). So that's the basic problem. As it's universal, i.e a problem across applications, its going to get better and information will be a lot easier to get access to...

As the answers are from the text, we can avoid hallucinations for the most part. I have not experienced made up numbers, instead errors are numbers that are typically misplaced, where it gives you revenues for the last 6 months instead of the last 3 months, when they are both next to each other in a table and so on... or the same numbers for different quarters and so on.

From my experience, GPT4 has been very good in following instructions and doesn't make up numbers which Bard is much more susceptible to. Bard relies heavily on snippets from the web search and completes the rest...

I am pretty sure if google wanted to train it, it would get all the answers right, but the way its designed, it gets one or two numbers and makes the rest up rather hilariously :) and even adds a breakdown which is also made up..

You can take a look here : https://imgur.com/a/vDxOV9D

Yes, I think the broader point is you have these readers available on call via api, so you can do all that you suggested and more...

I am using openai's latest functions api, so you can get it to return arguments that will ensure that you get a json, it works pretty well most of the time. The json would then be used to fetch a report from a database.

Embeddings did not really work well enough for me, let us say you are looking for numerical text, so for e,g get me AMD's revenue from March 2022. The embedding representation needs to understand that March 2022 together is way more important than March alone, I often ended up with March 2021 or March 2018 as being closer because the text might have multiple terms containing 'revenue' or multiple 'march'... Perhaps I could have improved it, but that did not seem like the right path to go down for accuracy.. This was way worse for e.g when I am looking for ECB statements, they can refer to an older date in their current report and it caused all sorts of trouble :). An initial fix was to basically mention March several times so that the search returns that... :)

I am using openai's functions as that is a more reliable form of extracting json, but even that can fail sometimes as the response misses a "'" or a }

You can try it out for yourself... :) Here's an example, that asks for AMD's cash and makes an arbitrary calculation on total liabilities, the ai is smart enough to sum up everything until equity and gets the numbers right, without any hallucination.

https://imgur.com/a/oAUZiIB

This is the source: https://www.sec.gov/Archives/edgar/data/2488/000000248823000...

The sum of all of these is 12831.

Total Current Liabilities 7572 Long-term debt, net of current portion 1714 Long-term operating lease liabilities 393 Deferred tax liabilities 1365 Other long-term liabilities 1787 12831

You are probably right, but there has been nothing like gpt4 before, having seen some versions before I would wager that their entire application can probably be redone for a fraction of the cost and it would work better... :)

I am not using inline XBRL, instead relying on gpt's ability to read tables.

I don't do handle news articles, and was just focusing on filings, so my answer would look for a filing. That's why it looks the way it does :)

Thanks, I don't feed the whole report in, unless necessary. Context size is a real limiting factor :), so you need to split it up into chunks that are meaningful on their own and find those relevant to the question. In this case, what are the current issues with the Disney company, This is how I would do it :) I get all the news articles related to Disney, this would then have sections (depends if you are using a json or a scraper), now you would then need to pick only those chunks and the surrounding ones (this would be an on the fly search) and then pass it to openai... This one is a very open ended question, this is what I get in beatandraise.com :) https://imgur.com/a/dqmcrRA

Thanks ! It's really interesting to see how LLMs have now brought back projects that were last updated years ago back into the limelight :)

I do not really imagine this as something that does all the investment work and makes a decision.

Instead, the mental model is you have an army of people who can read texts really well, as in 'reading comprehension' as they call it in english tests... This army can get you information on the fly.

Investment research involves a lot of back and forth reading and fetching tables and making conclusions, which in turn might not have much to do with stock price performance, but there's a whole industry of financial information and news for that :)

So currently, the scope is to make widely available information beyond what FactSet and CapIQ offer and even that's a long way away :)

This is how GPT4 answers the question... I assumed the question, but presumably will be along those lines :)

https://chat.openai.com/share/c736abf4-ae3c-4fbd-9427-b7d2f9...

Yes, that's right, I avoid hallucinations that way. In addition, I mark up existing texts so that the LLM knows what it is reading even if it is a piece of a larger text, so for e.g if you need to get Apple's Greater China sales for each quarter in 2020... https://imgur.com/a/oCCay3o

I am not making 4 calls with the entire text, I instead get pieces of each which would best match the question. There are additional challenges, for instance, GPT4 struggles to know the difference between the words guidance and outlook, which mean the same thing but somehow they don't for GPT4.

When I say they become better readers, I meant it in a general sense as in better in the case above. You basically have someone who can read through tables really well, and that can change investment research fundamentally, which is a lot of reading tables and graphs :)

Thanks, will try that.