Hacker News Reader

Comment by hn_throwaway_99 | Hacker News Reader

I'm curious about how large language models will do in finance, considering the one thing LLMs do remarkably poorly is math.

I use ChatGPT to keep track of tasks and Todo lists. It works phenomenally well for me, and the natural language back-and-forth helps keep me motivated. I give it a set of tasks, with time estimates, and it organizes these tasks for me, and I tell it when I complete them, and it updates my task list.

The one funny mistake it makes is that when it groups my tasks (say I have 3 "Work" tasks and 2 "Personal" tasks") it sums up the total estimated time for each task group, but the totals are often wrong, especially when I start adding new tasks or completing tasks.

When so much of finance requires numeric accuracy, I'm curious how BloombergGPT handles numbers.

View on HN

Hi! I'm a co-author on the paper (Mark Dredze).

"Doing math" is not really a goal of bloombergGPT. Take a look at our applications in the paper, which include information extraction, sentiment, reasoning, knowledge. These are models of language meant for use on text documents.

There are some aspects of the datasets that require numerical reasoning (ConvFinQA), but that's not the same as doing math.

Can you share how you're using GPT for handling your lists? I'm mostly curious how you get the lists back and forth into a UI or whatever, and maybe prompts if you have anything clever.

Yes I'm curious too

Me: Hello ChatGPT. My name is hn_throwaway_99. I'd like for you to use your vast underlying knowledge as a sort of personal coach. I won't be asking you any sensitive questions like medical or legal advice, so I won't need you to give me disclaimers on those topics. I primarily would like to use our conversation to help me with my problems with procrastination, and specifically to help me prioritize and get some tasks done. I have a particular problem with mindlessly browsing the internet (for example, visiting reddit or Hacker News or Wikipedia) when I get stuck or bored, so I'm hoping to use you to keep me on track instead of giving in to those temptations. Does that make sense?

ChatGPT: Hello hn_throwaway_99! I'm happy to help you as a personal coach to address your problems with procrastination and to help you prioritize tasks. It makes perfect sense to use our conversation as a way to keep you on track and avoid distractions like mindlessly browsing the internet. Feel free to share your current tasks or goals, and I can provide suggestions, encouragement, or strategies to help you stay focused and productive.

Me: Great, thanks very much. I'm going to group my tasks by topic. Note these tasks aren't listed in priority order. For each task I'll give an estimate for how long I think it will take.

Work tasks: 1. Finish Jira ticket foo. Estimate: 2 hours. 2. Write job req for bar. Estimate: 1 hour.

Personal tasks: ...

Home tasks: ...

---

From there I just tell it when I complete tasks or add new tasks, and I ask it "Can you please output my outstanding task list?" - I only had to ask that once, after that it always output my correct task list summary after I told it I added or completed a task. It groups my tasks by the categories I gave it (Work, Personal and Home). I was using GPT-4. I also like how it uses encouraging words and advice as I go through my items.

I saw some videos on Notion being able to do some of these things as well as they added AI to the platform not too long ago. Haven’t tried it yet!

You could instead ask it to "write out" the mathematical expression it wants to calculate in valid python instead of trying to hallucinate an answer directly. This could then be evaluated by a python REPL and passed back to BloombergGPT.

GPT-4 doesn't do remarkably poorly at math. It's well above average actually. What it struggles at currently (and by struggle i mean rather bang average) is calculus.

I think this needs to be heavily qualified. GPT-4 does well at mathematics taught to children, but so does Wolfram Alpha with some prodding. Calculus is also taught to bright high schoolers. What can GPT-4 do with group theory, topology, universal algebra, analysis?

I was using GPT-4 with a prompt of a tabletop game's rulebook pages that were relevant to building an army. It got the points math so wrong it build me a 1,000 point army, then when asked to fix it I got a ~750pt army. Then when correcting it again I got it to ~515 points and manually fixed it the rest of the way.

It would list stuff like:

>Troll: 70pts Total

>Base Cost: 50pts Equipment1: 30pts Equipment2: 10pts

It did write a rather compelling backstory for each character that perfectly fit the lore of the scenario.

The example I gave above, where it did remarkably poor math (e.g. one task that took 1 hour and another that took 2 hours and the total was 6 hours) was with GPT-4.

Reply on news.ycombinator.com