Multi-Agents LLM Financial Trading Framework | Hacker News Reader

Multi-Agents LLM Financial Trading Framework

github.com

If this worked it wouldn't have been open source?

Anyway, I have been running my own trading experiment and so far it has lost a bit of money. That being said I have not tried to optimise anything - just let it do whatever it wants. The losses are small and it might be able to recover later this year. Who knows.

The agent writes a blog about its progress here https://trades.chatbotkit.space/

I am thinking to output all the chat logs to HF as well for research.

You can run your own trading agents that communicate over a message buss in your own terms by downloading the CBK platform and running it locally with your own models. I have also shared my trading blueprint if you want to give it a go. https://chatbotkit.com/hub/blueprints/trader

Yeah, I agree with this. There is no point in open sourcing anything that works, in the finance world.

I actually have a profitable trading agent, and hell no I will not open source it. It also looks nothing like this. For one this bot is ingesting a shitton of noisy sources that have almost no signal and using them naively.

It would be more interesting to compare trading agents with index tracking ETFs. The better version of an ETF could maybe be a model where you zoom in on the companies and add/remove to your portfolio on the company related news, but keeping a broader portfolio.

Maybe agentic trading still performs worse than ETFs. But alternatively, if it were meaningfully better then it would be okay to opensource, similarly how ETFs are publishing their portfolios.

I think this might work.

When I started working no the trading agent I mentioned above I wanted to see if it can be just a better investor over the long run. The intention was not to do high-frequency trading. As you can see most of the days it is not taking any actions. The losses where down to mistakenly setting the stop losses too close to the top. If it wasn't so careful it might have made some money tbf.

My gut feeling is that AI agents will be able to manage a long-term portfolio much better than a human. Though it is just a gut feeling.

Lmao, you llm people have some crazy delusions. You realize markets are zero sum, and if you're using a public model that everyone else also has access too, you llm psychos will destory eachothers "agentic" edge (not that there ever was one). Not to mention all the other obvious flaws with llms, lime having an effective memory of ~200k words and no ability to judge whats actually going on in the real world.

The 'edge' is holding the investments over long periods of times. Agents are merely automating the portfolio managing part for lower costs.

Sure, but why would the organizations managing ETFs employ the same low cost agents + their own insights and provide a better return.

Yeah and you probably have to. An ETF easily has >1000 different stocks and even being weighted. So it has a completely different risk appetite by being so averaged.

But ETFs do have to follow particular rules defined by their product description. So it is still interesting to benchmark against.

Markets are only zero-sum in any given trade. Allocating capital to assets with higher growth rates (on the marginal dollar) creates value in the long run.

So, very simply, if AI can actually do better at picking a better long-term winner then it will increase growth.

You really have no idea what you are talking about.

I have been running an intermittent experiment with a multi agent "investment firm" for over a year now across model releases.

They certainly can beat indexes, BUT.. the model families have some biases that you have to design around. The stop loss that bit the parent is certainly one. The models like to create rules. Often rules, one of those is making all kinds of exit conditions.

Another big one from my experience is the bias to inaction in a scenario with risk. This means a model without structure around it will bias to keeping too much cash.

This would be nice if it was supported by some hard data. But even if it was, one year is too short of a timeline to make any kind of reasonable conclusions about its efficacy.

Buying stock based on coin flips can beat indexes short term too, that does not mean it is a better strategy or that it works over the long term.

There must be some room for some anti-llm agent that can profit from specific behaviors of these models when deployed against actual markets.

The idea that somebody here came up with idea that all professional algo traders didn't explore to the last penny a year if not more ahead of others is funny... but its not my money adding liquidity to the markets.

Is it opensource?

"just trust me bro"

Come to me when you have 500 trades and can beat Vangaurd's-VOO over a multi year time frame. Ill bet my entire networth and all future earnings for the rest of my life that your bot doesnt beat it. Your llm induced Dunning Kruger is going to get you in trouble one of these days I promise.

Look, it isn't fool proof and it is dangerous. With the current models you need to understand both markets and model biases and dynamics.

However with that said they are a huge multiplier and can tirelessly analyze the market for you.

They certainly can be used to beat sp 500 quite easily, but again that requires some understanding of risk on your part because the models will do what you ask them. If you go all in on options or something without clear risk management you will lose your ass.

The issue at stake is that financial markets are order-2 chaotic system, i.e. acting on them can change their outcome.

Put simply, if you open source your magic recipe, the behavioral change will affect the prices and you recipe will not work anymore.

The "blog" says that it's just holding cash or am I misunderstanding something? Judging by the "buying power" number, you allow it to use 4x leverage against the cash it has? That seems dangerous unless you are fine with losing 380k+.

A lot of people don't realize that when a leveraged position goes against you (especially with a regulated broker), you get liquidated once your equity runs out, but you can still owe the shortfall on top of that. And the broker can come after your assets to collect it. So your real exposure here is 380K, not 98K.

That being said, I think the parent AI is using paper money. Though who knows, this is the brave new world of AI.

If you like building but have no capital you might release something like this that works.

Am I reading these logs correctly? You gave an agent 100k!? Very bold, hopefully it does turn things around.

I had done a small experiment with 100$, but would be hesitant on more than that. I suppose if it's money you're willing to lose.

The title is 'Paper trades' so it's very likely fictional money.

Paper trades are more lenient afaik? You won't run into actual transaction problems particularly when market fluctuates by the milliseconds.

just seems to have trump tweets and war. surely those are nt the only market movers?

Having worked in hedge funds for the last decade, this seems to miss the mark. Firstly we often reward skillstacking ie a technical person later becoming a trader. The more one person knows the better. These people are rare though hence the reason there is still many seperate job functions, so a person can specialize. But an AI agent? They all have the same brain, so why nerf them by specialising.

Secondly, browsing reddit for sentiment and doing technical analysis is not even a feature in the trading world. At the most basic level, these are lagging indicators. Something on options IV and premiums would have been closer to the mark.

Hedge funds are akin to the maintenance crew for markets, we keep them efficient and liquid. The process is quite scientific, you come up with a theory and validate with real data. Or you go from data to theory.

> The process is quite scientific

This part is lost on many. The value of data is in the theories it confirms or more importantly disconfirms, and defining the trading edge is not easy after accounting for costs. I am wary of black-boxes that produce an edge - not only because I dont know how it works, but also because regimes shift unpredictably, and what works today may stop working tomorrow. That being said, AI can be useful to help automate many routine processes just like any other software.

Actually there is a really good reason, it has to do generally with making sure that all the aspects that you want evaluated are actually evaluated.

Now it may be possible with models like Astra that you no longer need to do this, but in earlier models it was beneficial.

So I might want a macro economic read which leads to a market thesis. Then I would hunt for exposure, then evaluate the candidates across different aspects. Breaking the process up at least made sure no steps were missed and the different aspects considered.

The obvious and dumb theory for trading on LLM recommendations is that lots of people are doing it so the price of stocks LLM recommend should go up. (ie: following the herd).

As local contexts/skills get better at depicting what's to be expected and what an agent can work with and work to get better at, having more and more little specialized agents working as a swarm get you, with a field-skilled human as a supervisor, really good results even in highly niche and technical fields

Reddit sentiment is mostly noise for a while now.

but...

options premiums imply volatility.

Lagging indicator would still be useful if it was accurate before the event. After the event, its just lagging history.

technical analysis is using historical data to make future predictions. No professional trader would trade anything without looking at price history.

Given what a know about the 2008 financial crisis, wouldn't an AI analysis in the years before that crisis of the real state funds helped to understand the risk of them better and avoid the big exposure.

Probably not. On the contrary, it would probably just amplify the mood.

The problem wasn't the analysis, given it was found out before it happened. The problem was politics, and as usual pushing the system to its limits and beyond.

I'd say it was more to do with how incentives lined up among different actors.

There's always risk somewhere in a financial system.

I spent about an hour looking at the code and found some glaring issues that should be fixed before trusting it with real money.

- Yahoo News is introduced twice (sentiment and news analysis) which double weights it

- Sentiment analysis prompt primes the model to be bullish on Nvidia.

- In the self learning loop there is a complex parsing bug that results in hallucinated memories when agents return truncated responses

- You can completely control sentiment analysis of a subreddit by simply maintaining a majority of the 5 most recently posted messages, regardless of any quality metric

- The reflection prompt states the agent must cite alpha, which in a market wide downturn causes it to think correctly placed calls were losses

> Sentiment analysis prompt primes the model to be bullish on

Eeh, yeah? At that point I'd stop reading the code and just leave the project behind. How exactly is the prompt doing this right now?

To be fair, these are not prompted as has been described here. These are actually "few shot" examples in the sentiment prompt. It is presented as an example under "distinguish opinion from event" but not as "do-this" evidence. Still, hard-coded positive Nvidia/NVDA examples in a generic prompt are unnecessary prompt contamination. At best this a real bias risk, but not a strong deterministic bug.

There is also problem with underlying models. There was a study, where they always repest the same investing/management strategy: trust strangers, be open minded/adopt to new unproven ideas, prefer cooperation... Basically they were trained on disney-boomer bull(shit) market of last 15 years.

They have zero guards against market manioulations, and will get wiped without bull market!

103K stars, so clearly it's popular. Has anyone here used it, and what are the outcomes like, and importantly, who is the target audience for this?

I can see the intention behind crawling social media and news feeds to determine some 'evidence', but am not sure if that's the best approach or even if an LLM is the best way to get an assessment, or whether having so many input sources is a good idea.

103K stars in a few months, smells of purchased stars IMO.

I feel like a lot of the stars for repos like this are from people who are riding the AI hype train and not actually interested in using the software

Actually you might be right, the number of issues for such a popular repo is extremely low.

It should be sponsored by a model provider.

It's always been a joke that stock prediction is a rite of passage for software engineers introduced to ML training. I see the same is true for commodity LLMs.

In case there is interest, I've got a fork with some custom improvements. See section "What this fork adds" in README.md

https://github.com/skanga/TradingAgents

I do not understand the value of multi agent approach? Isn't a single agent with a good harness better than any multi agent env?

It helps you spend more token, is more expensive, and thus is obviously more AI. Also novelty and more complexity means less scrutiny of the approach.

These are necessary and perfectly sufficient for an investment firm thesis I believe.

If it's a single agent then people will just call it a chatgpt wrapper, can't have that can we

Modularity maybe. Remember, agents are basically workflows that call LLMs in certain nodes.

As the saying goes, those who know don't say and those who say don't know.

Which will make you bankrupt faster, this framework or the its token consumption?

This would make sense only if the LLMs would be constantly updated with a new data set and training phase every day. In that case, I'd could see this approach as having some sense. But, otherwise, these are just stochastic machines trained on static (outdated) data and I don't see how their predictions should be better than any other method around or even better than a human guessing.

Using a token slot machine to beat a financial slot machine, what could go wrong?

I think multi-agent (eg _different_ underlying LLMs) everything is really the future. Code produced via multi-agent workflows and reviews seems noticeably better. I've been experimenting with a multi-agent message board recently: https://github.com/pjlsergeant/dogpark

I've built agents that call different LLMs and keep separated memories. Remember, agents are just long-running workflows with some nodes calling LLMs and that sometimes can be started as tool from other "agent".

There are times when I wonder if couldn't just draw then in a BPMN designer that allowed me to write custom code for nodes. Is BPMN still a thing?

Experimented with multi-llm analysis for problem solving over summer, combined with multi-agent approaches it can tease out interesting angles to problems that I never considered. Expensive but use only for my high value problems.

https://github.com/monkeydust/rightmind

There are good arguments for subagents/multi-agents. But the added overhead is usually massive.

Instead of going after money printing schemes like this (which I feel are unlikely to actually work and carry a ton of risk). I (thoughtfully) vibed up tooling and skills to just help LLMs manage my investments long-term. While not flashy, tax loss harvesting and direct indexing are areas where LLMs can reliably save you on fees that you'd otherwise have to pay. It's been very helpful, the LLM automatically deploys new cash based on my priorities, implements tax harvesting for me, and provides customized exposure guidance while keeping my goals in mind (ESG/sustainability focus, no FF rules) . Way better than any financial planner has ever done for me. https://github.com/mshafir/investbot

I vibe coded a little stock market sim game then i wrote an agent who’s job is to win the game. A couple friends and family members play the game too. If it works I’ll just follow along and hold the same portfolio the bot does. The bot, named stonker, makes its first trades in about an hour actually. Assuming it works I mean hah

https://stonks.jettdigital.app

edit: i just talked to my little sister ("Ginger" on the leaderboad) who is doing a good job beating the sp500. She said she just asks her ai what she should invest in and then executes those trades.

I’ve been running a day trading bot for the last 4-5 months, and it’s a lot of work. It never comes down to agent skills or abilities, more so the data you can receive and how quickly you can receive it. 1m bars, MFE calculations, rvol, executable bid/ask, spread, volume, market/sector context, and then making sure none of it has look ahead leakage

I’ve been doing a fairly similar experiment, but I ended up moving in almost the opposite direction than what this framework purports. deterministic code decides what is actually legal to trade, handles sizing/risk/execution, and an LLM (nanobot architecture) only gets to rank the already valid candidate set. If the model fails or times out, deterministic ordering takes over, so only the -nth degree of data actually makes it to the non-deterministic part (haha).

The hard part hasn’t been making the agents smarter/skillset but getting clean, fast data, preserving exact order/fill lineage (Postgres) and separating bad selection from bad execution or exits without leaking future information into the analysis

The multi agent debate stuff is interesting, but if every agent is reasoning over the same stale or incomplete inputs, I’m not convinced you gain much. I’ve built PoCs for my same project, and a round robin of LLMs is just hallucination and self approval city. Better data and tighter decision boundaries seem to matter more

100% of this comment is AI, against HN guidelines

https://www.pangram.com/history/8597362a-878d-4548-afb6-30fa...

100% is offensive to my sensibilities, and perhaps I do just write like a robot on the internet.

LLMs are notoriously bad with finance. I prefer deterministic rules for Agents like this one https://github.com/AlexBThomsen/vaultcharts-trading-engine, which I literally shared a few days ago on Hacker News, but no interest :D

What is the purpose of this repo? Is it to simulate the market so you can reliably backtest trading strategies?

Whatever the stated purpose is, where can I read the test results to show it accurately fulfills that purpose.

Anyone can make a markets simulation that models interactions between market participants. Making a simulation that is accurate enough to be useful for anything is hard.

I helped someone day trade 25 years ago and we made 40% in 4 months, mostly on AAPL, and it was easy. Way easier than working the 2 years moving furniture that it would have taken to earn that money, which I was foolishly doing to make rent for my defunct shareware business.

I've heard that high frequency trading eats market opportunities within milliseconds of spotting them now though, basically making the market even more random. The catch being that human nature isn't random, it's stochastic, so there will always be more money to be made on trades (or else trading firms wouldn't exist).

I'm thinking about getting back into trading because AI represents the end of buying software and we'll all be out of work soon, even if we're in denial about it. But everything I invest my time and energy into turns to crap. In a very real sense, as soon as I start trading, then karmically that could trigger recession, market reforms which ban what I'm trying, or even the end of money. I'm kind of joking and kind of not.

I've worked on a lot of really hard stuff over my brief but stupid career, and am basically used up mentally and physically, at least for now. Quant stuff is easy with AI. Should I try it?

No. If you could do it, with no remarkable skillset, that means people who are actually competent will be doing it orders of magnitude way better than you and you will be too far behind to really do anything significant except waste your time.

I'd be curious as to how correlated development on these frameworks (ai or otherwise) is correlated with the market cycle. It seems during bull runs would be traders think they have some edge - whereas they're probably all just buying the trend.

indeed. many strategies, agents and frameworks emerge during bull market.

https://github.com/ConsciousGroupMind/SKYNET-800---Collectiv...

I am sure it won't be long until some rando from /r/wallstreetbets/ turns something like this loose without considering the ramifications and loses $250k.

Losing that amount of money would be very uncharacteristic of wsb

How does this actually trade? It outputs investment decisions sure, but where's the interaction with an actual broker?

Anytime I see an investment framework I look for a “Performance” section or at least a backtest

There is a section in their paper.

Although I don't think it even matters. They could easily cherry pick a period that is favorable for them. Wasting time and money on short term trading, rather than long term investment, using LLM or not, is never a good strategy for most people.

They backtested with public models, wouldn't the model weights already have the data? I double checked with chatgpt and looks like agents also had web search tool available so they could just lookup the past.

https://chatgpt.com/share/6aa00d5e-6920-83ee-8e3e-9cbf23f7bd...

serious traders working on this type of project would have run extensive back testing on both real as well as synthetic analogues

not my agentic trading strategy but luck

I might contribute to this, my public github is looking stale for headhunters

1) costs? 2) profits/costs?

Play stupid games, win stupid prizes.

give them 1 dollar each run, and maybe you will make 2 bucks on the 100,000th run.

nightmare horseshit, don't waste your tokens

Reply on news.ycombinator.com