GPT-4 identifies SVB’s biggest risk & gives good advice using 2021 balance sheet | Hacker News Reader

GPT-4 identifies SVB’s biggest risk & gives good advice using 2021 balance sheet

blog.matteskridge.com

110 pointsnopinsight60 comments

A problem I have with this type of experiment is that it's almost pre-messaging the bot what to say. Simply filtering down to very specific variables leads the outcomes in a very strong way. I suspect this exact approach will not work well for finding future problems.

I would be more interested if the variables were not pre-selected with the knowledge of what was important and could determine things that haven't happened, not because of the prediction but because this means it would understand how to pick the variables that are relevant (like the human did in this experiment).

Anecdotally, the times I've felt most impressed by ChatGPT usually ended up being exactly like this.

Even as an LLM skeptic, I've had chat sessions with ChatGPT that felt unbelievable at the time, where it almost seemed to be generating incredibly cogent ideas and arguments all on its own. Then, after a few days, I went and reread the transcript, only to realize I'd been hinting and nudging it along much more than I consciously thought at the time. I only remembered the answers that were the most impressive and forgot about the ones that missed the mark.

ChatGPT is really good at picking out the key ideas in the prompt and responding to them. It's really easy to inadvertently nudge it to give a certain type of response, which of course makes it feel all the more astounding when it gives you exactly the type of answer you already unconsciously expect.

Indeed; I've been trying to describe these models as being consummate bullshitters. Or consummate improvisers, if you want to be kinder. Since its core task model is "autocomplete", it's less "trying to answer a question" than it is "trying to predict the most likely next thing to say after a question". Doing its best to sound like the most likely thing that comes after the thing prompting it, which in this case, often looks like an 'answer'. And when you have access to as much context as these models do, that can get as specific and nuanced as "giving you the answer you were expecting".

It has got me wondering if this might be an interesting fundamental model of consciousness. A bullshit generator modulated by other systems that may have more "logical" qualities.

I discuss this topic briefly in a post I recently made where I call it "decorative knowledge" and try to pinpoint exactly what GPT is doing and fails to do.

https://buildingbetterteams.de/profiles/brian-graham/navigat...

The AI equivalent of cold reading - just like with "psychics" where the audience wants to believe in the claimed abilities.

You must be psychic. I was literally thinking of cold reading as I wrote my comment.

It helps to think of it as augmentation rather than a separate entity. That way you can lead it consciously.

Talking to yourself is good for you as long as you know you are talking to yourself. You're training the social world model in your mind.

Eliza has come a long way, as Professors Higgins and Weizenbaum agree.

So ChatGPT is Clever Hans

Yeah my immediate response to reading this was "objection! leading the witness!"

You know it failed due to interest rate risk and you prompt it to score about interest rate risk.

It turns out that the bank that failed due to interest rate risk scored highly on it.

If you already knew to ask that question back in 2021, you would have known the answer as well, but neither you or the LLM are predicting the future accurately.

Perhaps someone could try other variations of the experiment as suggested above and by many here.

We should be running it on 1000 banks right now, and post the result next year.

Could you give an example of what other variables could’ve been included?

Elsewhere in this discussion thread there's a few examples from im3w1l who saw this problem too:

> Like it could have mentioned jobs numbers but it didn't. It could have mentioned covid hospitalization statistics but it didn't etc

"I was able to prompt GPT-4 in a way that makes it give information that is probably reasonable but that I had to then still verify myself, using my own expertise as baseline, because GPT-4 cannot be trusted to give true information, and it turns out that for this particular prompt the result made sense" is not something you should downplay.

This is exactly what I came here to say. Yes, GPT-4 may happen to give a correct answer. But what happens when it's wrong? And how would you know it's wrong without a specific background in some topic?

This is the major "magic wow factor" behind GPT. It has a basic level of knowledge in every topic which is enough to convince an average human that the AI produced a good result If-And-Only-If the human is unskilled in that topic.

Except that GPT can and does produce good results even to experts, just not all the time. So your claim of "IFF" is unnecessary hyperbolic contrarianism

The interesting part here is "does it take longer for an expert to come up with the answer independently or verify that GPT is producing an accurate result?"

The other day I was having ChatGPT write some code for me and it started referencing objects and properties that didnt exist. I wish I tried correcting it. That would have been interesting.

When you correct it, it always seems to say "You are absolutely right! I apologize for the mistake," and then it gives another wrong answer with the same fully confident tone.

Lol. One probably has to work at OpenAI to try and fix their LLM thing. The one they let the public use probably doesnt learn from its users. Hopefully. I wouldn't let just any data in my LLM. When those things get garbage in they just put garbage out.

There's a "thumbs up" and "thumbs down" next to each generated response.

While this model may not be getting updated in real time, I would be surprised if that feedback isn't used when looking at updating the model with good feedback being used for retraining the model.

I would love to hear opinions from LLM experts. But I have a feeling they may not be able to update the model, at a fundamental level, so easily by live feedback.

Based on what I have been reading, training the core model is a work of statistical analysis on how often words follow each other. This produces a graph tree of words.

Now how do you re-evaluate the entire graph based on a single additional feedback without actually retraining the entire model with the said feedback (because it is costly, and result wouldn't be immediate, unless you're thinking of injecting the input as an initial condition to the readily trained model)

It is possible to tune the model, and I suspect that remains something that is done on a regular basis.

https://platform.openai.com/docs/guides/fine-tuning

In particular from https://platform.openai.com/docs/models/gpt-3

> With the release of gpt-3.5-turbo, some of our models are now being continually updated. In order to mitigate the chance of model changes affecting our users in an unexpected way, we also offer model versions that will stay static for 3 month periods. With the new cadence of model updates, we are also giving people the ability to contribute evals to help us improve the model for different use cases. If you are interested, check out the OpenAI Evals repository.

The feedback wouldn't be immediately injected back into the model (human curation of the responses is needed to see if the feedback is appropriate).

Some of the feedback would be used to train the moderation / supervisor model. https://platform.openai.com/docs/models/moderation

How long was your token list by then?

Once context gets long with the limited tokenization memory we currently have it seems to go insane rather fast. Would like to test on the 32k model to see how the same prompting differs.

I thought I was using GPT-4, but it appears that's only available to paying customers so far? The question I was getting repeated wrong answers to was "Can you make up a palindrome that starts with 'Dude'?" If you want to try that with GPT-4 I'd be interested to see if it can do it now or at least knows to say "I can't".

It seems like the title is incorrect:

> GPT-4 identifies SVB’s biggest risk & gives good advice using 2021 balance sheet

From the article GPT-4 gave the likelihood of a bank run as a 1 out of 5. Yes, SVB, made some bad bets, but even with those bad bets they could have continued operating just fine. The thing that actually killed SVB was the $42 billion being withdrawn in a single day out of the $190B in total deposits.

The bad bets were the catalyst that triggered the bank run, but they were not the root cause. The root cause was SVB caters to a specific market segment, venture backed startups, that exhibit herd like behavior. All it takes is a few VCs telling their portfolio companies to move their money out of SVB and suddenly 22% of all deposits are withdrawn. I suspect few banks could survive a similar bank run. None of this is mentioned by GPT-4.

The bank run would be much less likely to occur had SVB diversified away from long-term bonds as GPT-4’s output recommended and thus avoided the huge losses on its books.

> From the article GPT-4 gave the likelihood of a bank run as a 1 out of 5.

.. what probability would you have assigned to a SVB bank run in 2021?

GPT-4 didn't say there's a 1 out of 5 chance of a bank run happening. On a 5 point scale from very low likelihood (1) to very high likelihood (5) GPT-4 gave a 1. GPT-4 gave the lowest score possible for how likely a bank run was to occur.

Of the six options GPT-4 evaluated, GPT-4 gave the lowest risk assessment code (combination of the probability of the event occurring and how bad it would be if it were to occur) to a bank run.

Bank runs are incredibly rare, rating it 1 out of 5 on likelihood it’s really overstating it by a LOT. I’m with the AI on this one.

How about a control or two? Would this model be able to see SVB as an outlier in a field of other banks?

Plus, given that sampling is probabilistic, one would have to prompt the model many times and evaluate all the responses. When I see examples as in the post, I always assume that the result was completely cherry-picked from a larger samples that contained all kinds of stuff.

I was expecting a broader type of question, rather than a series of more focused questions. I don't think the focused questions are "leading questions" necessarily, but it is sort of like "I have identified these risk factors, evaluate them." I would be more interested in the response to just an open ended "given this financial position what scenarios are likely to have a significant downside risk for the bank" or something like that.

For that matter, which banks are likely to be in a precarious financial position over the next one to two years and why? Even singling out SVB is doing a lot of heavy lifting. Once you know what to look for, it's a lot easier to find it.

It can operate at almost any level of abstraction.

So zoom out even further, to the intersection of macroeconomics and geopolitics.

Describe the big things that are happening in the world right now, from the banking crisis, to the inability to raise the debt ceiling, to social and political unrest, to disinformation campaigns promoted by crypto shills, to Russia's invasion of Ukraine.

Ask it to examine the relationships among multiple events in different domains interacting with each other.

Ask it to help with scenario planning by identifying multiple possible scenarios given that fact pattern.

Interesting and sobering results --> https://imgur.com/a/fL4CpkQ

I asked it the same question, but first requested risk factors followed by asking it to generate the scores for those based on SVBs balance sheet. This was the response:

“ To generate a risk score for the risk factors identified for Silicon Valley Bank, we will use a multiplication of likelihood vs impact. We will assign a likelihood score of 1-5 (1 being low likelihood and 5 being high likelihood) and an impact score of 1-5 (1 being low impact and 5 being high impact).

Economic downturn: Likelihood 4, Impact 4. Risk Score = 16

Technology industry disruption: Likelihood 3, Impact 4. Risk Score = 12

Concentration risk: Likelihood 3, Impact 3. Risk Score = 9

Credit risk: Likelihood 3, Impact 5. Risk Score = 15

Regulatory risk: Likelihood 2, Impact 3. Risk Score = 6

Cybersecurity risk: Likelihood 3, Impact 4. Risk Score = 12

Interest rate risk: Likelihood 3, Impact 3. Risk Score = 9

Using the balance sheet provided, we can calculate the impact of these risk factors on SVB's assets:

US Treasuries: 7.5% * Risk Score = 120

Mortgage-backed securities: 47.6% * Risk Score = 571.2

High-risk loans: 31.1% * Risk Score = 484.5

Cash: 3.7% * Risk Score = 44.4

Other asset classes: 10.1% * Risk Score = 161.2

The total risk score for all assets is 1381.3. This score is a rough estimate and is subject to change based on various factors such as changes in SVB's portfolio composition and the overall economic environment.”

I think it's a leading question. The initial paragraph mentions interest rates but none of the other factors, so it seems plausible that it took the hint. Like it could have mentioned jobs numbers but it didn't. It could have mentioned covid hospitalization statistics but it didn't etc.

I'm pretty sure any remotely competent financial analyst could produce something similar with presumably a lot more analysis of the likelihood of risk factors. (And hindsight bias would be cheating.) I also note that a run on the bank is perceived as very low probability.

Maybe it is. But perhaps we should be asking leading questions about every bank, and seeing what GPT comes up with.

Unless they can somehow guarantee that the model(s) were not trained on any data later than the 2021 balance sheet, then this is a completely useless exercise and is basically a case of "hindsight is 20/20".

The prompt sounds a bit like begging the question. I wonder what response it would give if it was worded more neutrally. For example, it's possible that leading with the current Fed funds rate biases it towards considering interest rate risk as more significant.

How do you know it doesn't use 2023 data just because you tell it to pretend its 2021?

The models have cutoff dates that are explained in the docs.

Why on earth would you trust that? OpenAI isn't releasing any technical details of how GPT-4 works. There may be some part of the model trained on pre-2022 data only, but there may be fine-tuning on later data.

Your view is the one with less evidence.

Has anyone tried to ask GPT to review an investment Portfolio and suggest areas of concern and/or improvements?

Personally as a noob to investing, this would be incredible. Obviously GPT can make stuff up but still.

>Obviously GPT can make stuff up but still.

Do you honestly think GPT is going to do a "better" (whatever that means) job than running scenarios with any of the many portfolio analysis tools out there? Particularly given it's a black box so you really can't trust what it tells you?

I guess we should browse over to WSB to see how much they've lost on doing this so far then.

If it works and everyone starts using it, then what do you think would happen?

That would actually be super interesting to see what would happen if everyone had a Homo Economicus in their pocket. We'd start getting into some good game theory territory

What could possibly go wrong with people using GPT for financial advice...

This could be useful for depositors to conduct KYB, Know Your Bank

The creators of these large language models (LLMs) are akin to our modern-day Einsteins and Teslas. They've harnessed the power of AI to produce truly transformative and seemingly magical advancements in technology!

Nobel prizes and MacArthur Fellowships are in order.

The trick now is to pose the same set of questions for every other bank.

That can tell you who is next.

Gpt4 is good if you not looking for anything novel.

In some ways "Thank god", this is already a disruptive product, and I'd rather have a moderately slow AI ramp up than have super powerful AI systems shake up the world.

It is amazing how these models are just not limited to step by step work but also can output better strategic decisions than many humans. It certainly seems more objective.

It is impossible to backtest an LLM.

Reply on news.ycombinator.com