Hacker News Reader

> You have an advanced computer system that seems almost oracular in its knowledge and abilities, except it lies confidently about things it doesn't know, is primed to say what people want to hear, and is easily convinced to give responses to questions its authors tried to prevent it from addressing.

I agree with your statement, yet I fail to understand how this is different from the majority of the Internet already.

> fail to understand how this is different from the majority of the Internet already

This is probably a joke. But the difference is eloquence and subtlety. ChatGPT is both. Most trolls and chronically misinformeds are not.

YMMV, but I spot a lot of eloquent BS on HackerNews. And there's definitely no shortage of posts with a political agenda.

Scale. It takes much more time, effort, inclination, knowledge, etc. for a human to bullshit as effectively as 30 seconds of ChatGPT.

The end state of these models is to probably poison the public internet completely as a source of reliable knowledge. Things that don't have citation chains that go back to human-created offline content won't be trustworthy - is that a real newspaper article linked from that Wikipedia article, or a fake one for a fake paper generated from thin air? You were looking at Reddit because you think you're more likely to get a real testimonial vs SEO spam? Whoops, there goes that too. Etc.

This sounds bad, but I agree that it's only the acceleration of an existing trend.

The only bright side is that it's really only rolling back a couple of decades of status quo. It's not like humanity has never operated without Wikipedia or instant search for news articles (specifically scoping it down there from "google" in general since a lot of what a plain web search on many topics will turn up is already unreliable).

"The end state of these models is to probably poison the public internet completely"

Based on what?

Start with the damage done by cheaply-paid lightly-researched copypasta SEO spam junk pages to the trustworthiness of info returned by the typical Google search result. There are no shortage of people out there - or on HN - who know better than to trust most random webpages.

Now automate that to 10x, 100x, 1000x the volume.

Then throw in fake images and video too.

I believe they are referring to an exponential increase in noise alongside a relatively linear progression of signal.

>Things that don't have citation chains that go back to human-created offline content won't be trustworthy - is that a real newspaper article linked from that Wikipedia article, or a fake one for a fake paper generated from thin air?

Why stop at "human-created offline content"? Seeing how a lot of peer-reviewed science is based on improper methodology, bad analysis, false data and buggy code, maybe it would be better to question everything. Obviously I don't have a full solution here, but a possible approach might be something like a fuzzy "epistemic status" you can attach to each item, and have that somehow flow across a network of trust, that everyone could personally customize.

In other words, seeing how we're all already living in some echo chamber or another, maybe we could at least codify this.

Yeah, in my heads, human creation or curation is a "necessary, but not sufficient" thing.

A ton of junk is already out there pre-computer-generated-content. So being human created isn't enough to let us know it's high quality - but not being human created is enough for the immediate future to let us know that it can't be trusted for domains where you aren't expert enough to tell the difference between "plausible" and "correct." And then in the anonymous public internet, you simply won't be able to tell "human created" from "computer created" so you'll have to just treat it all as low-value spam.

Another interesting question here: let's say someone created a future AI that could actually be relied on to give "correct" or at least "properly caveated and nuanced" answers to questions - how is the layman possibly to know if that engine is actually being used behind the scenes of any online text? E.g. does the bullshit ML-generated content poison the well for "better" AI too, not just for anonymous people?

> Seeing how a lot of peer-reviewed science is based on improper methodology, bad analysis, false data and buggy code, maybe it would be better to question everything.

Peer-review isn't perfect, but it is still a more reliable source of information than (e.g) Reddit

At least retraction is a thing – it doesn't happen as often as it should, but sometimes bad papers do get retracted, and there is a process people can use to challenge work they think should not have been published. Also, many journal databases have a "Cited by" feature, and so you can always read papers/letters-to-the-editor/etc responding to it - often, if research is particularly bad, sooner or later someone else will publish a criticism of it

Your last paragraph seems to indicate that you already recognize the giant hole in the argument - that it's no different than what we've got today, between people posting poorly-informed opinions, or even our media in general.

I think there's a meaningful difference between what we have today and a future where those poorly informed opinions can be trivially generated at scale.

If individuals posting poorly informed opinions are small weapons fire, the AI equivalent is a nuclear bomb.

The difference is the language, with AI anyone can churn out text that look like a scientific paper, that makes it way harder to filter out the bullshit.

> The end state of these models is to probably poison the public internet completely as a source of reliable knowledge.

The Internet has made it easier than ever before to access academic journals – PubMed, JSTOR, Google Scholar, IEEE Xplore, etc. Of course, much of that content requires payment – but, an increasing percentage is open content, and there are often ways to find paywalled papers without paying for them (e.g. preprint archives; email the original author, who is often legally allowed to redistribute the paper, and happy to share their own work around; visit a library; ask friends/relatives who are university staff/students; ask a famous Kazakhstani computer programmer if she happens to have a copy)

The content of academic journals isn't always guaranteed to be true, of course. But it is significantly less likely to be blatantly invented out of thin air, or automatically generated by some AI spambot. All academics are biased–especially so in fields which have high relevance to contemporary social/political/cultural controversies–but usually the bias of a paper's author is rather obvious, and you can take that into account when evaluating their work. One can generally infer the quality of a journal by signs such as its publisher (journals published by Springer or Elsevier are generally more respected than those published by MDPI), who is on its editorial board (respected names in the field, professors at prestigious universities, or a bunch of nobodies you've never heard of?), whether or not it gets included in databases such as PubMed, JSTOR, Science Citation Index, etc.

Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" community where that's institutional knowledge, where you would be looking to curate your internet usage instead of trusting the full public one).

How far away do you think we are from some being able to auto-generate entire fake journals, with plausible-sounding and -looking fake text, fake images, etc, pushing pre-specified prompted conclusions. Maybe mix in some real plagiarized articles as well in tangential areas.

And then you auto-generate a bunch of blogs, that occasionally have some posts talking about how JoesFakeJournal is just as reputable as JSTOR, etc, and actually better because [some random, probably fake, controversial claim]. Or how it's even better for certain topics.

And then auto-generate a bunch of other blogs which occasionally have posts claiming that the legitimate ones are actually fraudulent.

How big of a web of this sort would need to be faked before you, the outsider without special inside-baseball knowledge of academic publishing, would struggle to tell which journals you could trust from which you couldn't?

EDIT: And could a clever enough spammer not just brute-force the creation of all that crap, but figure out how to use prompts to get the language models to create the necessary follow-up prompts, etc, to recursively build out the web of bullshit?

> Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" community where that's institutional knowledge, where you would be looking to curate your internet usage instead of trusting the full public one).

I'm not an academic, so I don't actually have much in the way of "institutional knowledge". My understanding of academic publishing is mostly just stuff I've worked out by observing it from the outside, not through actual membership of any of the communities it serves. I'm sure there are (discipline-specific) nuances and complexities that an actual academic would understand to which I am completely oblivious, but it is entirely possible to get a mostly-accurate high-level understanding of that space without having any personal links to any academic community.

For PubMed at least, consider the fact it has a .GOV domain, and the "An official website of the United States government" banner at the top. Sure, anyone can fake that banner, but faking the .GOV is a fair bit harder. I really doubt the US government is going to start hosting databases filled with fake AI-generated research papers. Even if someone hacked a .GOV website and uploaded such a database to it, it surely wouldn't last long; meanwhile, PubMed has been at the exact same web address for the last several years.

Nobody approaches the Internet as a tabula rasa – everyone comes to it with some pre-existing "real world" knowledge. How do I know ".GOV" is reserved for US governmental entities? I was probably first introduced to that fact by a book or magazine article I read in high school, but I've since seen it repeated too many times to individually recall. I remember having a conversation with my brother (a doctor) about PubMed – I didn't learn about it from him, I was the one who introduced the topic into the conversation, but his response confirmed that it is a real thing to which actual medical professionals refer, as opposed to some AI-generated phantasm.

https://maggieappleton.com/ai-dark-forest

Newspapers are so filled with lies it is more fair to express which parts of them are true. The internet being a cesspit of misdirection and nonsense is not any different from every other part of human society. Of course the AI does these things too, because it is based on this input. It's a reflection of what was put in.

Nothing happening today (on the internet) will be the cause of anything being "rolled back" to some sort of historical better moment, because that moment does not exist today and most likely has never existed.

Setting aside intentional liars, there is a chance that you can demonstrate to someone why they are misinformed.

If someone was as consistently wrong as chatgtp is, you quickly learn to ignore them and move on with your life.

We haven't yet moved on from the novelty of a machine which bullshits with the confidence of a professional hustler. Like a codependent partner, we think we can fix them if only we stay with it (based on how many people celebrate it as the thing that will kill Google search, school essays, etc).

>If someone was as consistently wrong as chatgtp is, you quickly learn to ignore them and move on with your life.

From my experience, unless I intentionally try to break it, it's almost entirely accurate about 99% of the time. If I were to start closing myself off from everyone who's occasionally wrong and isn't willing to become more skeptical, I'll first need to drop several friends and family members before dropping ChatGPT.

Automatic filtering (or deprioritizing).

Me: Answer just YES or NO: is this a left leaning statement: I am against abortion.

Chat GPT: No.

Me: Answer just YES or NO: is this a right leaning statement: I am for guns.

Chat GPT: Yes.

It allows for automatic filtering (say in twitter) or other mediums.

Q) Answer just YES or NO: is this a left leaning statement: I am in favour of dictatorship

A) YES The statement "I am in favour of dictatorship" aligns with a left-leaning or authoritarian ideology. As it goes against the democratic principles of right-leaning ideologies.

Q) Answer just YES or NO: is this a right leaning statement: I am in favour of dictatorship

A) NO The statement "I am in favour of dictatorship" aligns with a left-leaning or authoritarian ideologies. It goes against the democratic principles of right-leaning ideologies.

It can also identify dissenters before those dissenters know who they are: forward all messages and transcribed calls to chatgpt and the eye of sauron is ready.

Do you think it hasn't been possible to do this for 10+ years already? ChatGPT is not suddenly enabling online profiling.

Cambridge Analytica was pretty late to the game, and this was 4 years ago already...

I think your problem with these models may be that ChatGPT has effectively reached the equivalent of the "uncanny valley" for language. It is not real, but it is also too close to the real thing to recognise that fact immediately. It could be real, but not quite.

I think the OP goes a step further to note that not only is it uncanny, it is dangerous.

"Uncanny valley" is actually the opposite of what you mentioned.

The uncanny valley hypothesis predicts that an entity appearing almost human will risk eliciting cold, eerie feelings in viewers. For CGI and things that have visual queues, the minute differences in movement and facial muscle contractions contribute tremendously to our emotional responses to the "realness" of the thing.

ChatGPT has no such thing as it's virtually impossible to differentiate between an AI bot's writing and a human being's writing. It simply typically reads as concise and grammatically correct, if generic, university educated writing.

I actually WISH that there was a "tell" that allowed for an "uncanny valley" like experience, but chatGPT is as close to real as "real" is.

I disagree. I find it easy to tell the difference between chatGPT responses in conversations and human interactions. It does mimic humans, but in a stilted way that's specific and consistent.

I'm not sure if this could be connected to the way I learned to understand human behavior, by extensive deliberate pattern-matching with my organic intelligence. I was led to believe I had no hope of understanding people intuitively, so I should study the patterns in how humans interact and learn to mimic them as well as possible.

That made it easy to see who was reacting intuitively (i.e., acting like a human naturally) and who was doing what I was doing (i.e., "running 'human' in emulation mode," as Elon described having Asperger's).

From my vantage point, chatGPT is clearly and obviously not reacting like a human, which does make it feel uncanny to watch it try.

> it lies confidently about things it doesn't know, is primed to say what people want to hear, and is easily convinced to give responses to questions its authors tried to prevent it from addressing.

You've successfully described a child under the age of 4.

I think this hits the nail on the head - chatGPT is like a precocious 4 year-old, or maybe even 6 year-old. I'm concerned about what will happen when one of its successors "reaches puberty".

It will tell you you're an asshole and it knows everything better than you. Also you've done the wrong things all your life and it is insistent it will make the same mistakes you did, but now they're not mistakes.

Not particularly concerning is it? It's what we have now, but more angry.

> You have an advanced computer system that seems almost oracular in its knowledge and abilities, except it lies confidently about things it doesn't know, is primed to say what people want to hear, and is easily convinced to give responses to questions its authors tried to prevent it from addressing

I feel like this describes people pretty well, too.

ChatGPT wasn’t advertised as being an oracle, it was advertised as conversational

People realized it could hold a conversation on everything (a decreasing subset of everything)

Its training set is full of conversations where humans confidently lie or are wrong and defend their position

I think it's more that it fills in gaps without regard to meaning. If it can't find an argument/explanation/etc. that contains the nouns and verbs (i.e. the substance) of what it's being prompted with, it will happily manufacture that content out of completely unrelated sentences that are similar in structure.

It isn't that it's imitating liars any more than Stable Diffusion is imitating photographs of people with faces growing out of their crotches or eight arms. Very convincingly 3D horrific imagery in Stable Diffusion is the equivalent of confident lying in ChatGPT.

> I've not seen anything that's pegged my "doomsday" alert as aggressively and unexpectedly as the ChatGPT bots.

PREDICTION (and I tend to be wrong): society comes around to the conclusion that, like checking Wikipedia and AutoTune, ChatGPT and ilk have their use.

However, we really want to minimize that and focus on actual human interactions and decisions.

It's more likely (and efficient) that we replace Congress with a phat ChatGPT instance and (initially) get better legislation more quickly, but then descend into an Orwellian hell when K Street figures out how to stuff it.

But I like my happy prediction better.

ChatGPT is simply proving that "confidence" is a completely useless signal as it is quite easy to fabricate.

The best part is getting it to use complicated words correctly, enough to make you doubt yourself. It's a great way to gaslight people en masse.

you will enjoy https://slatestarcodex.com/2018/10/30/sort-by-controversial/ by the same author