Ask HN: Is the weaponisation of ChatGPT now inevitable? | Hacker News Reader

Ask HN: Is the weaponisation of ChatGPT now inevitable?

43 pointsusgroup55 comments

I'm not sure whether its possible to obtain this model from OpenAI or how hard it may be for unscrupulous actors to train their own on the same principles.

On the assumption that it is entirely possible, wouldn't the natural consequence -- amongst others -- be weaponisation for unscrupulous purposes? Won't it result in 1000s of blogs mushrooming over night from purported bots-come-experts with some kind of engineered slant? Whole "online communities" for the unsuspecting to join initially made up of nothing but bots "talking" to each other, bots written journal papers / editorials, etc, and so on.

What comes next?

View on HN

My rough take is that the internet as we know it (web 1.0 or 2.0 depending on your taste/age) is over.

It ended for me when I found myself in the uncanny valley moderating a Facebook group (while not personally being on Facebook). A user was acting very bot-like. I went to ban them but thought I should look into what the account was first and found conversations they had with others, who were also probably bots given the strange shallowness of the discussion. The uncanniness was that “real” Facebook conversations seemed strange and shallow to me already and I couldn’t rule out this account being real. In that moment though, I also couldn’t know for sure whether I’d ever seen a real conversation or human on Facebook, or most darkly, the difference didn’t matter because the real people on a platform or medium amenable to “bot-ification” become themselves “bot-ified“ and speak just like them. I’m old enough to remember feeling and talking about the shallowness of internet text messaging, and heard those conversations in my head again.

From the excitement of connecting with interesting people in the 90s, this was a dark turn for me. And now that World Wide Web of interesting people is dead. We may retain special guarded fortresses and hopefully manage to keep them alive. But also, maybe good riddance. If the internet evolved in a monstrous direction, maybe it wasn’t worth it in the first place or it’s at least worth letting go …

I can see this as positive — the web 2.0 idiocy of social media is going to disappear, and we'll hopefully see more smaller groups with more interesting conversations. HN is leading the pack, and this is the only place I really go to anymore.

Web 2.0 products like Linkedin, Twitter, and even newer ones like Nextdoor are optimizing for terrible interactions.

I miss the old 90's kind of forum interactions, where civilized, deeper conversations reigned. I think the only way this can be done is with smaller, more "aligned" communities, and strong moderation tools, team, and policies.

This can be accomplished while maintaining profits at scale. I've been thinking of about this problem since 2009. The current paradigm never worked.

The reason why HN usually doesn't have the banal conversations that are common elsewhere is due to moderation. ChatGPT makes dan g (and other HN mods') jobs much more difficult.

I mean we're already basically there, and have been for a while now. There are lots of spam SEO sites that already just aggregate and repackage content in extremely low-value, barely-coherent ways. Not to mention the huge quantity of real people who spout nonsense and untruths in their personal blogs or on forum posts.

Using the internet at all is an exercise in filtering out what seems relevant and credible from all the rest of the garbage -- and it always has been. It's why we rely on major news brands, trusted personalities, and other high-reputation sites like Wikipedia, etc. And one of the main functions of search engines from the very start has always been to try to direct you to high-quality content over spam (which is an arms race and the engines aren't 100% accurate but they're certainly pretty good).

So I don't really see ChatGPT having much material impact at all in this department. It's somewhat similar to Photoshop in this regard -- whenever we see a photo these days, we're aware of the fact that it might be entirely fictional. But we've got enough sense to know that if it's a news photo in the New York Times it's probably real, while if it's something coming from a random Twitter account without any history of credibility, we should assume there's just as much chance of it being fake.

The internet has always been swamped with fake/garbage stuff, and adding more isn't going to change much, because we just continue to use the same critical thinking and skeptical eyes we've always used, evaluating which sources are actually credible.

OpenAi just recently created a Diplomacy player. Not just managing the pieces on the board, but also negotiating with other players--all in natural language. The programmed ended up in the first few places of a large tournament, but amazingly none of the other players had catched on to the fact that they were playing an AI.

It seems then that nothing is to stop such a program from interaction to further other goals. Even if an audio conversation is beyond it, it can send me messages on my Instagram, LinkedIn, and what have you. It may try to convince me of whatever message the highest bidder wants me convinced of. That this may not succeed in all cases, or even most cases makes little difference. Gaining a 5% advantage will sway most elections.

Think about that for a second. From now on: whenever you talk to someone online, that you haven't vetted in real life before, you cannot be sure that it is an actual person. Soon, it will be likely that it is not a real person.

You just made me realize the first level tech recruiters are over as a job. Many of them use blind templated reach outs that will actually be better with AI looking at your exact profile and GitHub.

> OpenAi just recently created a Diplomacy player

For any wondering, the paper is Human-level play in the game of Diplomacy by combining language models with strategic reasoning https://www.science.org/doi/10.1126/science.ade9097

You’re thinking of meta: https://ai.facebook.com/blog/cicero-ai-negotiates-persuades-...

ChatGPT is the death knell for discussion communities where mainstream internet users spend their time. That would be Reddit, Quora, Facebook and Twitter.

We don't need to ring that bell for search engines, it was already drowning in Wikipedia co-authored blogspam that this won't change much.

"Dead internet theory" posits that bot-generated comments and content is more common than we think. Reddit actually has subreddit simulators that are nothing but bots talking back and forth to each other, using training models based on actual Reddit comment threads. It's not chatGPT level, but it's frightening to see how bland most 'real' Reddit conversations are.

do you have a link to any of the good botspam subreddits (besides worldnews), I mean intentional/declared botspam ones?

I don't have a Reddit link, but here's a Hackernews thread where there's only one human comment and the rest are bots: https://news.ycombinator.com/item?id=33952087

I see what you did there.

You're just linking to the current thread?

Information grey goo.

Why do you think that only "mainstream" communities will be impacted? I've seen a lot of HN comments that seem pretty clearly machine-generated, even a few special-interest Discord servers I'm in. At this point, the only people I'm sure aren't LLMs are people I know personally in meatspace.

https://xkcd.com/810/ comes to mind: if I can't really tell, I guess it shouldn't really matter. You might be a bot, but I still had fun composing this reply to you, or I wouldn't have done it.

The only danger comes in trying to use online conversations to gauge "public opinion" incorrectly, but this was already impossible to do because of the the now-old-fashioned Filter Bubble problem. Really, the LLM spam problem is just a new frontier in filter-bubbling: computers can now generate a unique filter bubble just for me!

By mainstream, I mean those with the largest aggregated audiences. That's where the people are, so that's where the money will be. The paid, polished, 'enterprise' version of ChatGPT will need to pay for itself after all.

I am sure bot content is far more common than we think since we seem to have a bias that we are always talking to someone on the other end of the message.

I think though we are really underestimating how much we will prefer the chat bot at some point.

A language model just has to approach some % of the human you most respect on a topic to listen to in order to dominate the want to talk to hundreds of random reddit humans.

It is just such a low bar to pass. Seems only a few years at most and I will just not be bothered to read things that haven't been generated by a language model online because 99% of the human output will be shit in comparison. It won't be worth the time to even look for that 1% of human output that isn't shit because the language model will be close enough and easy to find everytime.

We've had regular GPT for several years now, and nothing has happened. People predicted this with the OG Transformer model from OpenAI, and yet the fake news apocalypse didn't come upon us.

> Won't it result in 1000s of blogs mushrooming over night from purported bots-come-experts with some kind of engineered slant?

I don't say this to be rude, I really don't, but have you Googled something like "reverse a string python" at any point in the past 5 years?

> Whole "online communities" for the unsuspecting to join initially made up of nothing but bots "talking" to each other,

This has also existed for a long time, see SubredditSimulator etc. There are also sites that scrape forums and replace the usernames to do ad fraud.

You could carry on a decent conversation with ELIZA in the 90s. ChatGPT is arguably less sophisticated since it's just using NN tricks rather than trying to emulate intelligence. I could be wrong but I think ChatGPT is just about the ceiling for what this technology can do. I don't mean to be a grump because it's incredibly impressive for what it is but I still think we're pretty far from a tipping point.

I think the big innovation here isn't in its grasp of language, but in the way it's able to access data. Comparing ChatGPT against ELIZA or GPT-2 isn't the right comparison, you should compare it to something like SQL or Google.

Against that comparison, it stands out as very innovative, but it also seems like it has big potential to improve. If you could perfectly index a large database and carry out reasoning tasks, this is enough to replace a lot of jobs. ChatGPT is in this regard just on the boundary between a toy example and useful in prod.

It's also worth noting that, up until now, people who have been saying this ("I think this is about where it peaks, it'll never do X") have always been proven wrong by the passage of time. So I think we should assign lower credibility to the skeptics, insofar as their past predictions were incorrect.

What do you mean by "the way it's able to access data"? AFAIK ChatGPT is just a fine tuned version of GPT-3.5 optimized for chat[1], it isn't any different from GPT-2 in the way it accesses data.

[1]: https://openai.com/blog/chatgpt/#methods

That's true, but since the model is bigger than GPT-2, I can basically use it as a query language for a database.

If the model is smaller, I still do that, sure, but I can only ask "basic" questions about the English language and so on. But if it has more neurons, it's capable to "index" a lot of questions about e.g. proper Python usage.

It's definitely made several huge leaps in ability. Not least of which is just compiling and processing the enormous corpus of training data. It's also definitely a major leap in NLP. Not just in terms of tech, but also UX in that it can funnel users into the relatively narrow set of prompts that it can understand and respond to perfectly. By comparison, Alexa is ten years old and barely understands half the things I ask and frequently goes way off topic in it's responses. Admittedly a very different product but in the same space.

I think that there's a real risk that "information becomes useless" in the AI era. Neither text, speach/sound, images nor videos can be trusted, and it's far toi easy and rewarding for unscrupulous actors to use AI to put out fake/biased news/comments/opinions/media.

This has probably been said many times before - but it also seems like a new era for digital surveillance or espionage: bots could "befriend" humans, create full networks of fake people crafting any story imaginable in order to extract information. Maybe

More possibly, the domain most improved by real-time English chat will be phone scams.

luckily for us, most have learned to not even answer the phone after the past 5 years on unending call spam.

I was thinking "recruitment for cults / terrorist groups." In 5 years, whenever someone seems lonely on the internet, a "friend" will magically materialize and sympathetically listen to their problems. Gradually the "friend" will start turning the conversation to their goals. ("Oh, your girlfriend left you? That sucks... but do you know who's REALLY to blame for all your problems?")

And there are a lot of lonely people out there...

This isn't likely IMO. It's not like there's a robust API to find "lonely people" in the first place, and the difficulty in creating sympathetic people to talk to for a long period of time without slipping up once will be immensely hard. What if they want to voice chat, or play a video game?

Also, don't cults have a massive surplus of people-time as is? I get the feeling that, if I wanted to go and join a cult, it's not like they would tell me they were short-staffed and there was a waiting list.

More likely (and mundane) outcome is that people use it to create cheap online therapy services. Good ROI, clearly defined goals, very large addressable market, easier to find people willing to work on it.

Well, I hope I'm wrong! ;-)

My guess is that it'll be like "Nigerian prince" scams: it's not all that convincing. But even if it only works on 0.1% of its targets, that's still a large absolute number of people.

That would also be the case for just spamming people with "go join our cult" links, and I haven't seen this once. The limits load on other things:

- Local presence of the cult

- Ability to operationalize victims (convince them to do terrorism, in your model)

- Ability of victims to "function" in society (hold down a job, do cult administrative stuff, earn money)

For an idea of what is the upper bound, look at Scientology or Jehovah's Witnesses. They don't have to use an AI to talk to people, they just hand out fliers on the street and talk to whoever is interested. Lonely people already today can sign up for Scientology, so the AI adds nothing.

Yes, ALL tech eventually gets use for good and bad. It follows human will.

Even in its current state, people are figuring out how to use it for their purposes. There's no need to train it or get any more info on how it works. What we see and get is enough.

I suspect that in a few years from now the internet will be flooded with AI crappy content. I wonder how that's going to influence future AI? Since, the future models will be trained with content created by AIs.

When the internet is flooded with crappy AI content, people will move to walled gardens even more, since they can afford the cost of moderation.

All it takes is one bot to get in the walled garden though and it is the same.

It is not exactly the same though. To torture this metaphor a bit more, it is easier to weed and prune a walled garden than it is to maintain a commons. Effective gatekeeping makes it a bit harder or more expensive for attackers to join communities and this also makes moderation of these invaders more costly for them, so they have to spend more time/effort to maintain the masquerade if they do not want to waste the expenditure required to enter the community.

LinkedIn is pretty walled, and it's a cesspool. What prevents Brian Jones, a 100% legitimate human being, from pasting in a few GPT-generated posts to improve his posting activity?

At the end of the day, you have to filter on quality.

I wonder if there could be some standard at the device level with copy/paste and signing/verifying that you did not or the source from where you did copy from?

Well, what if I generate some nice GPT-based content on my personal computer, look at it on the screen, and then re-type it by hand on my "protected" smartphone?

All you end up doing is implementing another ham-fisted DRM scheme that ultimately accomplishes nothing but makes the pointy-haired bosses happy. Solving social problems like this with "moar technology" is, in the general case, not a recipe for success, and I don't see why this would be any different.

true, such an interesting problem. you're right, maybe more like a social solution than strictly tech.

Which is hard, when you're looking for an answer: something you don't know. How do you tell what's "quality" and what's not?

LinkedIn is actually better off than most sites, because it already has a "people you actually know" filter built in. If you've connect to randoms, you'll get random content. But if you connect only to people you know, you have a decent filter. And if people you know start posting rubbish, block them.

Any pseudoymous sites are in a worse position, because you can't tell a person from a bot.

LLM is incredibly useful as an answer engine*. I can imagine this replacing the Google search box on every mobile device. Why search, browse, decline tracking popups and newsletters signup popups, and then try to fish out scraps of information out of a SEO fluff piece, on a tiny screen, if the LLM can tell you the answer straight away, and answer your follow up questions.

This can be a serious threat to search engines, and by extension information-providing parts of the web, and web advertising (you will be getting ads from your LLM provider, and everyone on the web will hate that their content has been hoovered up for free).

Then we'll have spam trying to manipulate data used to train the bot. There will be commercial spam, but also political/ideological spam. And here's the scary part: to reject the spam, people training those AIs will have to compile a big table of what is true. This will guide the AI that people listen to, and which can be an excellent bullshitter. There is a massive potential for abuse here.

*) the current one is too often confidently incorrect, but that is probably fixable to a good-enough level.

I believe the baseline expectations will simply increase like what Seth Godin said about this:

> If your work isn’t more useful or insightful or urgent than GPT can create in 12 seconds, don’t interrupt people with it.

https://seths.blog/2022/12/attention-trust-and-gpt3/

I've asked GPT to write stuff for me related to my job and none of it was useful. It's amazingly good at staying on brief and giving cogent and coherent responses but it is still utterly incapable of thinking. And it's also incapable of staying up to date. It's very impressive for what it is but idk whose job it might be taking any time soon.

OpenAI added stricter rules this week-end (given all demos rely on re-using the session token from their public demo):

https://github.com/transitive-bullshit/chatgpt-api/issues/96

The current version available at https://chat.openai.com/chat seems neutered to the point of being both less useful and less interesting than just searching google.

If they don't do that people goad the model into recommending we eat glass or similar ideas.

Surely we shouldn’t throw the baby out with the bathwater, no? Trying to control this is surely futile — how long until a company like Stability AI releases an open source, uncensored version of GPT?

I respond with a maxim: 'Anything that can be used for surveillance will be used for surveillance' Shoshana Zuboff

I would also apply this to the case of AI - such as those for understanding whether or not someone will commit a crime

What would be the point of content farms if google is basically replaced, I won't visit your blog to learn about weevils I'll just ask chatGPT, and never leave the interface at all.

This already exist. ChatGPT perhaps increases the scale of the fakery, but doesn't introduce a new concept.

"Quantity is a quality of its own."

We are back in the Altavista days. We need a new Google, maybe a local one this time.

Of course it is inevitable. Humans are involved.

It seems only inevitable that ChatGPT will be used to exploit children.

Reply on news.ycombinator.com