Hacker News Reader

Comment by tsuujin | Hacker News Reader

I find this to be really concerning if true (which I’m not convinced of, seems more like correlation than causation).

The most amazing part of chat ai is its ability to be very convincing and yet factually incorrect. It doesn’t seem like it’s a secret that these chat apps spit out just plain wrong information routinely, I’m not sure why anyone would trust it with something important.

SO has bad info too, for sure, but part of SO is peer review to help feel that out. “AI” is a black box and not to be trusted.

View on HN

I tried asking ChatGPT how to do something for the first time yesterday (an example of how to use libtar to write a tar file from buffers in memory, in C). It gave a very confident, but spectacularly wrong answer, concocting a whole new (admittedly more convenient for this purpose) API for libtar. After that experience, I'm not sure I'll ever ask ChatGPT for help again...

ChatGPT is (currently) like having an incredibly fast junior engineer working for you. You can ask it to do tasks for you and it will do them quickly, but you need to check its work. Sometimes it will be perfect, sometimes it will be right with a little more prompting, and sometimes it is completely wrong.

Yep. But how does it compare to GPT-4 or Bing Chat? Then wonder how well GPT-5 will perform, and 6. Also, did rubber-ducking your query have a benefit regardless of the correctness of chat's answer?

You use 3.5 or 4? It’s worth paying for 4. Significantly better results.

I think the premier use for ChatGPT right now is as a starting place for writing tasks. I use it to give me a template for work emails from time to time because there is no element of correctness to get wrong.

I give it facts I want to see in prose, it gives me very neutral, inoffensive prose back. I always have to issue a few corrections to the prompt, and I always have to post edit, but it still saves me a huge amount of time and I don’t need to worry so much about bias leaking in and making anyone uncomfortable.

I think it’s likely that, unless regulated, commercial LLMs will have a chilling effect on most human-created information sources. With eliminating the opportunity for authors to engage with information seekers (a chance to get constructive feedback, upsell something, show ads, see interest stats) will come reduced willingness of authors to publish.

If you publish on StackOverflow only for LLM to gobble it up and no human even sees the upvote button anymore… why would you? You are basically an unpaid ghost content producer for Microsoft/ClosedAI at this point.

It opens an opportunity for shilling, pushing your agenda, and other ways of poisoning users' generated content. While LLMs will be brainlessly sucking in and replicating. Till they learn to check first and filter out obvious fakes. Still there will be not obvious. The problem is how to make money on this...

I wonder if the very nature of LLM emergent correctness and intelligence itself will identify weakness, fallacies and illusion? I would expect a well-read person to be similarly immune.

Humans aren't immune to manipulations. Neither personally, not in mass. LLMs should be possible to manipulate to some degree if attacker controls significant portion of information on some topic. For example manufacturers can 'upgrade' specs on their devices. Till they get caught LLMs will parrot them as a ground truth. The same with 'human values' and political events interpretation. China already introduced policy on LLMs, they should be politically correct. LLMs producers can manipulate, but not eliminate, the bias by selecting and weighting the sources. Which they do. Plus humans interventions when they see unpleasant output.

Humans have free will and ethical standards and ability to manipulate ideas. LLMs have none of that. They are tools designed to do what told.

This is a simplistic vision. First, not everybody believes they have no consciousness, or ideas. Or cannot generate new ideas. This already happened with 'crazy' chess gameplay style. Humans had never seen it before.

Yes, they are tools, build by humans. But this doesn't mean humans can understand what, how, and why they are doing. Historically observations may take hundreds of years to explain.

If it is admitted to be conscious, idea-understanding, it would be enough like a human and would have to have human rights (and not abused like it currently is). If it doesn’t have human rights, it’s not human-like enough for this to matter. I think these are mutually exclusive.

> “AI” is a black box and not to be trusted.

Someone will be the first against the wall when the revolution comes. I know treason agains Friend Computer when I see it.

Now if you’ll excuse me, I have some hallways to paint red.

You should try Bing chat. It provides citations for almost every assertion. Allowing you to get a quick summary/conclusion, which is also quickly verifiable if you want.

https://www.phind.com/ is also similarly good, but for dev related content.

I find it best to use it with it framed as an untrustworthy black box, even with the increasingly correct answers. Its a rubber duck that can spitball with you.

Reply on news.ycombinator.com