Hacker News Reader

Comment by verdverm | Hacker News Reader

Notice how the actual data on these pages is exactly the same and from 2022-2023?

This is what I mean by making up links that don't work. The links further down the page don't even go back that far.

You ought to inspect the links before posting them as proof. By not doing so, you demonstrate the limitations and fallacies of humans putting to much faith in these tools.

This graph made 7 years ago would appear to contradict the numbers ChatGPT gave you: https://www.reddit.com/r/dataisbeautiful/comments/3s3c8o/rel...

Are you ready to rethink your exuberance for ChatGPT?

---

I performed two searches, both from image search, took 1 minute to find a good graph. I suspect ChatGPT took longer to write the combined, inaccurate responses, based on my experience at how slow it streams the answers back.

1. windows version usage by year, 2000 - 2010

2. operating system usage by year, 2000 - 2010

View on HN

I challenged it on the statcounter figures, and this was the response:

----

I apologize for the confusion. You are correct that StatCounter Global Stats only goes back to 2009. I must have misremembered.

Here are the worldwide market share figures for Windows 98 and Windows XP from NetMarketShare, which provides historical data for desktop operating systems:

Year Windows 98 Windows XP

2002 60.3% 0.6%

2003 45.3% 19.2%

2004 21.8% 50.0%

2005 8.2% 63.9%

2006 3.4% 76.1%

2007 1.6% 74.7%

> I must have misremembered.

The fact that ChatGPT says this is very misleading. It does not perform recall, just token generation based on the most likely continuation of the input. It will even finish your questions

Without this thread, how do you know enough to challenge the results? If you have to challenge every output, why is it useful in the first place?

It's useful for some things but not others.

For example, I used it to help me know how to adjust some things with matplotlib that I hadn't used in years. I wasn't even sure what to type into google to get good results, looking over the docs would have taken longer than the time I spent with ChatGPT.

Like how we all learned how to search with Google early on, the same thing is going to happen with LLMs. They are certainly better at handling the questions and clarification for refinement, something we often do with each other. When they start responding with questions for clarification, rather than just spitting out whatever, that will be a nice step forward.

Are you even checking the sources it claims? NetMarketShare only looks to go back to 2016. :facepalm:

At this point, your methodology is looking quite sus

Are ready to accept that ChatGPT is reliably lying to you? The longer your conversation with it goes on, the worse it gets

But the numbers seem credible and realistic when you think about OS usage at that time. If it was hallucinating from web-pages that only go back to 2009, how on earth did it come up with such credible figures?

It's not looking things up, it is making things up. This is what it was designed to do... that's why they call it "generative AI"

Half of statistics are made up, so the saying goes. It's easy to make "credible and realistic" numbers if you understand confirmation bias. You want to believe in ChatGPT, so you see its numbers as credible and realistic, despite them not being so.

Also understand that we are in the beginnings of a hype cycle. There was a time when HN was exuberant about the potential for blockchain. I expect that our expectations for LLMs to be made realistic with time. I also believe that LLMs will provide value, unlike blockchain, so the hype cycle is probably closer to dotcom in how it plays out.

The plugin system will be interesting to see play out. It's sort of a hybrid system, using LLMs as an interface for intent while calling out to other systems for accurate retrieval. I'm curious how far it can be pushed, it's hard to see these things being able to interact and navigate all of the tools we have created. Too much variability and opaqueness.

> You want to believe in ChatGPT, so you see its numbers as credible and realistic, despite them not being so.

I didn't really, I had doubts myself. I say the are credible based on my own understanding of how OS usage changed, being there myself at the time.

> I say the are credible based on my own understanding of how OS usage changed, being there myself at the time.

This is a classic example of a confirmation bias. How would you know the statistics from that time? Based on anecdotal observations from 20 years ago?

It's not really. I don't know the exact figures, but it seems like common sense to think WinXP would have very low usage in Jan 2022, given it was released in Oct 2021. And anecdotal observations after living through a time and using computers a lot seems like a reasonable basis to gauge the rough correctness of stats.

not sure I am following you any more, but pretty sure XP is way older than Oct 2021

Your experiences were biased by being someone technical, windows was widespread, I know of a place still running XP as recently as Oct 2021 tho... there are some business that don't take updating seriously, so what you saw is not evidence of the general population.

Back to the point, it's best not to rely on your feelings when it comes to judging ChatGPT's accuracy

Reply on news.ycombinator.com