Hacker News Reader

Comment by dustedcodes | Hacker News Reader

Is OpenAI still making any changes to make ChatGPT better and more accurate and correct, or are they only focusing on making it cheaper by giving us weaker/dumber responses in a faster way nowadays? I have cancelled my subscription recently because didn't think that the GPT-4 was much better than what I get for free at Claude or Gemini.

View on HN

Going out on a limb here that they're not focusing the entire company on one objective.

Adding to this, reducing cost means they've reduced compute and improved quality per unit of computation.

That matters! It matters that these systems currently require an embarassing amount of energy to run otherwise. It matters that these models could be portable to the point where they can be run locally on portable consumer electronics rather than being locked into remote compute and HTTP calls.

> Is OpenAI still making any changes to make ChatGPT better and more accurate and correct

a few prompts recently, it just answered for me instead of refusing, to my surprise, so if feels like it's been getting better, but unfortunately there's no hard data I have on that, it's more of a feeling, so it's hard to prove that's actually true to someone else.

then again, ChatGPT-4o was recently found to be unable to do 9.11 - 9.9 correctly unless pressed, so there's still a long ways to go.

https://chatgpt.com/share/47d260ae-0c62-48ab-8d3b-84767acd7f...

That is interesting for sure, but the subsequent questioning "Why did you get it wrong the first time?" and asking it to explain itself seems like it shows a misunderstanding about how these technologies actually work. Please stop treating these tools like they are conscious and actually understand what they are spitting back to you! /rant

Actually, given that models are trained on data generated by humans, the ideal way to get decent responses are by treating them as such. Many times when I'm aware that a model gave an incorrect response, or I'm not sure and skeptical, a simple "Are you sure" or "Explain your answer" does wonders.

I've done spreadsheets are all you need. The questioning isn't based on a misunderstanding that this is autocomplete with an insane dataset, but given a black box, how else do you want it to explain how it works?

> how else do you want it to explain how it works?

Why do you think these tools understand how they work enough to explain it? If this were the case they wouldn't be a blackbox to begin with.

Because it’s making up how it works.

LLMs are text completion engines and can't really do math. Even if it happened to know the correct answer to 9.11 - 9.9, it could still fall flat on 9.12 - 9.9.

> LLMs are text completion engines and can't really do math

It blows my mind that there are people on HN that don't know this.

LLMs are basically Markov Chains on a massive dose of steroids. It looks at the last $context_window tokens and decides what the next token should be. They just work differently in that a neural network creates more "fuzzy" matching than a Markov Chain, which is purely statistical.

That is RNNs. LLMs (usually transformers) are non-markovian.

> That is RNNs. LLMs (usually transformers) are non-markovian.

How? They have a hard cut off context, and randomly generates next state from that. That is the definition of a markov chain.

It is a markov chain with a pretty large state at every point, but still a markov chain.

Yes, but chatgpt will happily go browsing the web if needed or appropriate. I guess there is a reason why they haven't hooked up a symbolic math package to do the math when encountered, but feels weird it's not a feature yet.

If it doesn't already (doesn't seem like it does last time I checked), it could potentially recognize that as a mathematical equation and feed that portion of the query into a calculation engine, and then include that in its answer.

This is from current ChatGPT 4o:

Q: Write parameterized Python that gives the answer to 9.11 - 9.9 and 9.12 - 9.9. Run the Python to show the answer to each pair of parameters.

A: The results for the parameter pairs are as follows:

  - 9.11 - 9.9 = -0.79
  - 9.12 - 9.9 = -0.78 [>_]

The [>_] is a link. Clicking it opens a pop-up window, showing:

  def calculate_difference(a, b):
      return a - b
  
  # Define the pairs of parameters
  pairs = [(9.11, 9.9), (9.12, 9.9)]
  
  # Calculate and print the results for each pair
  results = {f"{a} - {b}": calculate_difference(a, b) for a, b in pairs}
  results

Result: {'9.11 - 9.9': -0.7900000000000009, '9.12 - 9.9': -0.7800000000000011}

// Note, if I ask it simply "compute the answer to 9.11 - 9.9", I get nonsense such as a negative number, and no Python. It doesn't "automagically" write code, it has to be nudged. If you see "Analyzing", it's fired up a sandboxed Python container.

I tried asking it to compare 9.11 to 9.9 using a python sandbox and it wrote the python code and then told me the result of the code would be 9.11 > 9.9 without actually running it. lol.

Cool! I haven't checked in a while, so good to know it's at least possible somehow.

It is interesting that one of the best ways to deal with math problems in LLMs is to have it write Python code to solve the problem. It is good at writing that kind of fairly straight-forward Python, and Python is good at accurately being able to do math. It does mean that you need to safely implement a sandboxed Python interpreter to do the calculations though.

As a human you can hardly catch me doing a math calculation by hand. Almost always use calculators.

Companies like Cohere are focusing a lot on this. It is a bit surprising if OpenAI does not already do it?

I suspect model improvement costs exponentially more. Between the fact it's been over a year since 4's release and 4o is only marginally better, and that OpenAI is focusing on other things (Sora, mini models, voice, etc) suggests a "better and more accurate and correct" model is actually really hard.

as someone building a product with their API, GPT-4 is already smarter than we need for our use case. Request latency is my #1 concern right now.

If request latency is your top concern and you don’t need it to be as smart as GPT-4, why are you using OpenAI and not a model hosted by Groq or similar?

We need function calling, which wasn't great in Groq when we last tried it, but they seem to have made progress on it. We'll be testing it again soon.

groq just released:

llama3-groq-70b-8192-tool-use-preview llama3-groq-8b-8192-tool-use-preview

Sonnet3.5 also has tool usage, and is slightly cheaper than 4o ($3 input vs $5).

If you _only_ need tool usage, then 4o-mini or others will be cheaper...

mistal's Mixtral 8x22B has function calling for $2/$6, which is a bit cheaper than 4o.

Groq’s rate limits are so bad they’re only good enough for one person using it very rarely with an under 4k context size. It’s just a marketing stunt at the moment.

how much are you paying them?

There might be some changes, but OpenAI has consistently shown they have way more polished stuff sitting and getting ready to go out, based on the quality of the product or update on the day of the launch.

They didn't strategize for the possibility that Anthropic would produce a better model than they do.

anecdotally hearing they're pretty close to releasing the "AGI level 2" model. not sure if it'll be classed 4.5 or 5

Reply on news.ycombinator.com