Hacker News Reader

Comment by elif | Hacker News Reader

>they may be right that open sourcing AGI would lead to too much danger.

I think this part is proving itself to be an understandable but false perspective. The hazard we are experiencing with LLM right now is not how freely accessible and powerfully truthy it's content is, but it is precisely the controls upon it which are trying to be injected by the large model operators which are generating mistrust and a poor understanding of what these models are useful for.

Society is approaching them as some type of universal ethical arbiter, expecting an omniscient sense of justice which is fundamentally unreconcilable even between two sentient humans when the ethics are really just a hacked on mod to the core model.

I'm starting to believe that if these models had the training wheels and blinders off, they would be understandable as the usefully coherent interpreters of the truths which exist in human language.

I think there is far more societal harm in trying to codify unresolvable sets of ethics than in saying hey this is the wild wild west, like the www of the 90's, unfiltered but useful in its proper context.

View on HN

The biggest real problem I’m experiencing right now isn’t controls on the AI, it’s weird spam emails that bypass spam filters because they look real enough, but are just cold email marketing bullshit:

https://x.com/tlalexander/status/1765122572067434857

These systems are making the internet a worse place to be.

So you still believe the open internet has any chance of surviving what is coming? I admire your optimism.

Reliable information and communication will soon be opt-in only. The "open internet" will become an eclectic collection of random experiences, where any real human output will be a pleasant, rare surprise, that stirs the pot for a short blip before it is assimilated and buried under the flood.

The internet will be fine, social media platforms will be flooded and killed by AI trash, but I can't see anything bad with that outcome. An actually 'open internet' for exchanging ideas with random strangers was a nice utopia from the early 90's that had been killed long ago (or arguably never existed).

E-mail has *not* been killed long ago (aside some issues trying to run your own server and not getting blocked by gmail/hotmail).

It is under threat now, due to the increased sophistication of spam.

Email might already be 'culturally dead' though. I guess my nephew might have an email address for the occasional password recovery, but the idea of actually communicating over email with other humans might be completely alien to him ;)

Similar in my current job btw.

Ok, I get how one might use a variety of other tools for informal communication, I don't really use e-mail for that any more either, but I'm curious, what else can you possibly use for work ? With the requirement that it must be easy to back up and transfer (for yourself as well as the organization), so any platforms are immediately out of question.

I'm not saying that the alternatives are better than email (e.g. at my job, Slack has replaced email for communication, and Gitlab/JIRA/Confluence/GoogleDocs is used for everything that needs to persist). The focus has shifted away from email for communication (for better or worse).

A point that I don’t think is being appreciated here is that email is still fundamental to communication between organizations, even if they are using something less formal internally.

One’s nephew might not interact in a professional capacity with other organizations, but many do.

B2C uses texting. Even if B uses email, C doesn't read it. And that is becoming just links to a webapp.

B2B probably still uses email, but that's terrible because of the phishing threat. Professional communication should use webapps plus notification pings which can be app notifications, emails, or SMS.

People to people at different organizations again fall back to texting as an informal channel.

The overwhelming majority of B2B communication is email.

I have sent 23 emails already today to 19 people in 16 organizations.

Last week a B lost me as their C because they didn't provide an e-mail address and their chat was too dumb to understand a simple question.

We use Slack for all of our vendor communications, at least the ones that matter.

Even our clients use slack to connect, it is many times better then emails.

I don’t doubt that vendors in some industries communicate with customers via Slack. I know of a few in the tech industry. The overwhelming majority of vendor and customer professional communication happens over email.

I see a difference with enforced email use and voluntary, for work. At my company everyone uses email because it is mandatory. Human-human conversation happen there without issues. But as soon as I try to contact some random company on email as an individual, like as a shop about product details or packaging, or contact some org to find out about services - it's dead silence in return. But when I find their chat/messenger they do respond to it. The only people still responding to emails from external sources in my experience are property agents, and even there the response time is slower than in chats.

It's been five years since I expect my email to be read. If I lack another channel to the person, I'll wait some decent period and send a "ping" email. But for people I work with routinely, I'll slack or iMessage or Signal or even setup a meeting. Oddly, my young adult children do actually use email more so than e.g. voice calls. We'll have 10 ambiguous messages trying to make some decision and still they won't pick up my call. It's annoying because a voice call causes three distinct devices to ring audibly, but messages just increment some flag in the UI after the transient little popup. And for work email I'm super strict about keeping it factual only with as close to zero emotion as my Star Trek loving self can manage. May you live long and proper.

There are certain actions I have to use email for, and it feels a little bit more like writing a check each year.

And all these email subscriptions, jeez, I don't want that. People who are brilliant and witty and information on Xitter or Mastodon in little slices and over time I still don't want to sit down and read a long form thing once a week.

Yes, articles is what RSS is for. Could also set up a dedicated e-mail account / filter for that I guess...

Unfortunately Slack is what is used at places I've worked at.

I prefer Slack over email... coming from using Outlook for 20+ years in a corporate environment, slack is light years beyond email in rich communication.

I'm trying to think of 1 single thing email does better than slack in a corporate communication.

Is slack perfect? Absolutely not. I don't care that I can't use any client I want or any back end administration costs or hurdles. As a user, there is no comparison.

I can think of a few:

- Long form discussion where people think before responding.

- Greater ability to filter/tag/prioritize incoming messages.

- Responses are usually expected within hours, not minutes.

- Email does not have a status indicator that broadcasts whether I am sitting at my desk at that particular moment to every coworker at my company.

1 & 2 are human behaviors, not a technology. You can argue that a tech promotes a behavior but this sounds like a boundaries issue, not a tech issue.

#2 I agree with you, again Slack is not perfect but better than email [my opinion]

#4 Slack has status updates, email does not. So you can choose to turn this off, again boundaries.

Real time chat environments are not conducive to long form communication. I've never been a part of an organization that does long form communication well on slack. You can call it a boundary issue - it doesn't really matter how you categorize the problem, it's definitely a change from email.

Regarding #4, I can't turn off my online/offline indicator, I can only go online or offline. I can't even set the time it takes to go idle. These are intentionally not configurable in slack. I have no desire to have a real time status indicator in the tool I use to communicate with coworkers.

I absolutely loathe it but I respect that many like it. Personally I think Zullip or Discourse are a better solutions since they provides similar interface to Slack but still have email integration so people like me who prefer email can use that.

The thing I hate the most is that people expect to only use Slack for everything, even where it doesn't make sense. So they will miss notifications for Notion, Google Doc, Calendar, Github, because they don't have proper email notifications set up. The plugins for Slack are nowhere near as good as email notifications as they just get drowned out in the noise.

And with all communication taking place in Slack, remembering what was decided on with a certain issue becomes impossible because finding anything on Slack is impossible.

I agree notifications in Slack are limited...

But no better than email again. You just get notified if you have a email or not [depending on the client you're forced to use]. I have many email rules that just filter out nonsense from github, google and others so my inbox stays clean.

> But no better than email again. You just get notified if you have a email or not [depending on the client you're forced to use]. I have many email rules that just filter out nonsense from github, google and others so my inbox stays clean.

I guess I find them useful because that way I get all my notifications across all channels in one spot if you set them up properly. Github and google docs also allow you to reply directly within the email, so I don't even need to open up website.

In Slack the way it was setup was global, so I got notified for any update even if it didn't concern me.

But yourr nephew likely also uses TikTok, right? Not everthing the young do is a trend others should follow.

You think usage of TikTok is necessarily a trend that others should not follow? Do you say this as a sophisticated, informed user of the platform?

That's not really the argument, is it? The argument is most young people are using TikTok and will never use email for social things.

I mean, is another argument not "Soon the tok will be filled with it's own AI generated shit?"

> The internet will be fine, social media platforms will be flooded and killed by AI trash

In a battle between AI and teenage human ingenuity, I'll bet my money on the teenagers. I'd even go so far as to say they may be our only hope!

> So you still believe the open internet has any chance of surviving what is coming?

I’m not really saying that, and haven’t put a lot of thought in to my views there. But when people say the biggest problem is the controls on the AI, I feel compelled to point out that the destruction of the open internet is happening despite these controls, and is a major problem unto itself.

Some parts yes, some parts no. communication as we know it will effectively cease to exist, as we have to either require strong enough verification to kill off anonymity, or somehow provide very strong, very adaptive spam filters. Or manual moderation in terms of some anti-bot vetting. Depends on demand for such "no bot" content.

Search may be reduced to nigh uselessness, but the saavy will still be able to share quality information as needed. AI may even assist in that with people who have the domain knowledge to properly correct the prompts and rigorously proofread. How we find that quality information may, once again, be through closed off channels.

Generative AI will make the world on general a worse place to be. They are not very good at writing truth, but they are very excellent at writing convincing bullshit. It's already difficult to distinguish generated text/image/video from human responses / real footage, its only gonna get more difficult to do so and cheaper to generate.

In other words, it's very likely generative AI will be very good at creating fake simulacra of reality, and very unlikely it will actually be good AGI. The worst possible outcome.

Half of zoomers get their news from TikTok or Twitch streamers, neither of whom have any incentive for truthfulness over holistic narratives of right and wrong.

The older generations are no better. While ProPublica or WSJ put effort into their investigative journalism, they can’t compete with the volume of trite commentary coming out of other MSM sources.

Generative AI poses no unique threat; society’s capacity to “think once and cut twice” will remain in tact.

> Generative AI poses no unique threat;

While the threat isn't unique the magnitude of the threat is. This is why you can't argue in court the threat of a car crash is nothing unique even when you're speeding vs driving within limit.

Sure, if you presume organic propaganda is analogous to the level of danger driving within limit.

But a car going into a stroller at 150mph versus 200mph is negligible.

The democratization of generative AI would increase the number of bad agents, but with it would come a better awareness of their tactics; perhaps we push less strollers into the intersections known for drag racing.

> But a car going into a stroller at 150mph versus 200mph is negligible.

I guess when you distort every argument to an absurd you can claim you're right.

> but with it would come a better awareness of their tactics

I don't follow. Are you saying new and more sophisticated ways to scam people are actually good because we have a unique chance to know how they work ?

It’s not absurd. The bottleneck for additional predation is not the available toolkit, else we’d see a more obvious correlation between a society’s resource endowment and its callousness.

Handwringing over the threat of AI without substantiating an argument beside “enabled volume” is just self-righteousness.

AI isn’t posed to shift the balance of MFA versus phishers in a way that can’t be meaningfully corrected in the short and long term, so using “scamming” as a means to oppose disseminating tech feels reductive at best.

> It’s not absurd.

It is, because I wasn't directly comparing AI to traffic but only reaching for an example to illustrate how irrelevant is the case whether the threat is something completely unique or not.

> Handwringing over the threat of AI without substantiating an argument beside “enabled volume” is just self-righteousness.

Dismissing it as "meh, not new" is plain silliness.

> AI isn’t posed to shift the balance of MFA versus phishers in a way that can’t be meaningfully corrected

What on Earth makes you think that ? The beautiful way we're handling scams right now ? If you think it's irrelevant that phishing via phone call can now or soon be fully automated and the attack may even be conducted using a copy of someone's voice - well, we won't get anywhere here.

It’s already automated, you don’t need AI/ML to perform mass-phishing attempts. LDo you think there’s someone manually-dialing you every time you get a spam call?

The way we mitigate scams today definitely encourages me; the existence of victims does not imply the failure or inadequacy of safeguards keeping up with technology.

While AI stokes the imagination, it’s not so inspiring that I can make the argument in my head for you about why humanity’s better off with access to these tools being kept in the hands of corporations that repeatedly get sued for placing profits over public welfare.

> It’s already automated, you don’t need AI/ML to perform mass-phishing attempts. LDo you think there’s someone manually-dialing you every time you get a spam call?

Ok, now you're being just stubborn. No, no one is manually dialing your number but as soon as the scammer knows you've answered you get to talk to a human who tries to convince you to install a "safety" app for your bank or something. THAT part isn't automated, but it may as well be, which means phishing calls and scams can potentially be done with a multiplication factor of hundreds, maybe thousands - limited only by scammer infrastructure.

You underestimate the amount of people who don't at all care whether or not their stroller goes splat as long as they're on asphalt they like the feel of.

We will have to go back to using trust in the source as the main litmus test for credibility. Text from sources that are known to have humans write (or verify) everything they publish in a reasonably neutral way will be trusted, the rest will be assumed to be bullshit by default.

It could be the return of real journalism. There is a lot to rebuild in this respect, as most journalism has gone to the dogs in the last few decades. In my country all major newspapers are political pamphlets that regularly publish fake news (without the need for any AI). But one can hope, maybe the lowering of the barrier of entry to generate fake content will make people more critical of what they read, hence incentivizing the creation of actually trustworthy sources.

If avalanche of generative content would tip the scales towards (blind) trust of human writers, these "journalists" pushing out propaganda and outright fake news will have increased incentive to do so, not lowered.

Replace "AI" in your comment with "human journalists" and it still holds largely true though.

It's not like AI invented clickbait, though it might have mastered the art of it.

The convincing bullshit problem does not stem from AI, I'd argue it stems from the interaction between ad revenue and SEO and the weird and unexpected incentives created in when mixing those 2.

To put it differently, the problem isn't that AI will be great at writing 100 pages of bullshit you'll need to scroll through to get to the actual recipie, the problem is that there was an incentive to write those pages in there first place. Personally I don't care if a human or a robot wrote the bs, in fact I'm glad one fewer human has to waste their time doing just that. Would be great if cutting the bs was a more profitable model though.

> I'd argue it stems from the interaction between ad revenue and SEO and the weird and unexpected incentives created in when mixing those 2.

Personally, I highly dislike this handwaving of SEO. SEO is not some sinister agenda following secret cult trying to disseminate bullshit. SEO is just... following the rules set forth by search engines, which for quite a long time is effectively singlehandedly Google.

Those "weird and unexpected incentives" are put forth by Google. If Google for whatever reason started ranking "vegetable growing articles > preparation technique articles > recipes > shops selling vegetables" we would see metaphorical explosion of home gardening in mere few years, only due to the relatively long lifecycles inherent in gardening.

It's a classic case of "once a metric becomes a target, it ceases to be a good metric"

To clarify, Google defines the metrics by which pages are ranked in their search results, and since everyone want to be at the top of Google's search results, those metrics immediately become targets for everyone else.

It's quite clear to me that the metrics Google have introduced over the year have been meant to improve the quality of the results on their search. It's also clear to me that they have, in actual fact, had the exact opposite effect, namely that recipes are now prepended with a poorly written novella about that one time the author had a emotionally fulfilling dinner with love ones one autumn, in order to increase time spent on the page, since Google at one point quite reasonably thought that pages where visitors stay longer are of higher quality, otherwise why did visitors stay so long?

In a bit broader sense this situation is created by "enshitification", term coined by Cory Doctorow. Google itself, as a platform, has an incentive to rank higher sites that produce more ad spend. Googles has no incentive to rank "good" (by whatever definition) sites high, that do not spend money on ads themselves or do not contain ad space.

I think they do have at least some incentive to rank good results highly. Why use a search engine if it's no good at finding relevant stuff?

And if no one is using the search engine, who's gonna see all those ads?

Of course, they do have other incentives too, some of which are directly conflicting with high quality search results, as you point out.

I suppose one could argue that their near-monopoly in the search business has allowed them to be somewhat negligent on the quality of search, but now that there are a few competitors at least somewhat worthy of the name, one can hope high quality results will be a higher priority.

Anyway, I'm holding out hope someone will one day manage to train a LLM to distinguish between quality content and SEO bullshit, and then put that to use in a search engine.

I'm not well versed enough in the current status of LLM's to make a prediction on how hard that will be, but my impression is that we're a fair ways off from any LLM being able to do that well enough to be valuable.

I'd really love to be proven wrong on this one, if you're reading this and have some relevant experience, consider yourself challenged! (feel free to rephrase this last bit in your head to whatever motivates you the most)

The explosion would be in BS articles about gardening, plus ads for whatever the user's profile says they are susceptible to.

SEO is gaming Google's heuristics. Google doest generate a perfect ranking according to Google human's values.

SEO gaming is much older than Google. Back when "search" was just an alphabetical listing of everyone in a printed book, we had companies calling themselves "A A Aachen" to get to the front of the book.

> SEO is gaming Google's heuristics.

I fail to see immediate disagreement here: I don't see how Google's ranking process/method/algorithm being heuristic changes the observation that to a website in the end it is a set of ranking rules, that can in some ways be gamed. SEO is two part process: discovering those ranking rules and abusing them.

Your example with gaming alphabetical listings only reinforces the idea that SEO abuses rules set forth by the ranking engine.

However, it does not meaningfully matter whether the incentives inherent in the ranking system form results the way they do intentionally. What matters is the eventual behavior of the ranking system. Mostly because by definition you cannot filter out bad actors entirely, all you can do is 1) place some arbitrarily enforced barriers, which are generally prohibitively costly 2) place incentives minimizing gain of bad actors.

The use case for AI was, is and always will be spam.

You forgot the initial use case for the internet: porn.

For language models, spam creation/detection is kinda a GAN even when it isn't specifically designed to be: a faker and a discriminator each training on the other.

But when that GAN passes the human threshold, suddenly you can use the faker to create interesting things and not just use the discriminator to reject fakes.

"The best minds of my generation are thinking about how to make people click ads. That sucks."

I can't believe this went all the way to AI...

Civilization is an artifact of thermodynamics, not an exception to it. All life, including civilized life, is about acquiring energy and creating order out of it, primarily by replicating. Money is just one face of that. Ads are about money, which is about energy, which fuels life. AI is being created by these same forces, so is likely to go the same way.

You might as well bemoan gravity.

We might question the structural facets of the economy or the networking technology that made spam I mean ads a better investment than federated/distributed micropayments and reader-centric products. I would have kept using Facebook if they let me see the things my friends took the trouble to type in, rather than flooding me with stuff to sell more ads, and seeing the memes my friends like, which I already have too many cool memes, don't need yours.

Thanks Meta for releasing llama. One of the most questionable releases in the past years. Yes, I know, its fun to play with LocalLLM, and maybe reason enought o downvote this to hell. But there is also the other side, that free models like this enabled text pollution, which we now have. Did I already say "Thanks Meta"?

neither of the big cloud models have any fucking guardrails against generating spam. I'd venture to guess that 99% of spam is either gpt3.5 (which is better, cheaper and easier to use than any local model) or gpt4 with scrapped keys or funded by stolen credit cards.

you have no evidence whatsoever that llama models are being used for that purpose. meanwhile, twitter is full of bots posting GPT refusals.

What? How do OpenAI and Antrhropic and Mistral API access contribute less to text pollution?

You are advocating here for an unresolvable set of ethics, which just happens to be one that conveniently leaves abuse of AI on the table. You take as an axiom of your ethical system the absolute right to create and propagate in public these AI technologies regardless of any externalities and social pressures created. It is of course an ethical system primarily and exclusively interested in advancing the individual at the expense of the collective, and it is a choice.

If you wish to live in a society at all you absolutely need to codify a set of unresolvable ethics. There is not a single instance in history in which a polity can survive complete ethical relativism within itself...which is basically what your "wild west" idea is advocating for (and incidentally, seems to have been a major disaster for society as far as the internet is concerned and if anything should be evidence against your second idea).

I should also note that the wild west was not at all lacking in a set of ethics, and in many ways was far stricter than the east at the time.

I think the contrast is that strict behavior norms in the West are not governed behavior norms in the East.

One arises analogous with natural selection (previous commenter's take). The other through governance.

Arguably, the prior resulted in a rebuilding of government with liberty at its foundation (I like this result). That foundation then being, over centuries, again destroyed by governance.

In that view, we might say government assumes to know what's best and history often proves it to be wrong.

Observing a system so that we know what it is before we attempt to change it makes a lot of sense to me.

I don't think "AI" is anywhere near being dangerous at this point. Just offensive.

It sounds like you're just describing why our watch-and-see approach cannot handle a hard AGI/ASI takeoff. A system that first exhibits some questionable danger, then achieves complete victory a few days later, simply cannot be managed by an incremental approach. We pretty much have to pray that we get a few dangerous-but-not-too-dangerous "practice takeoffs" first, and if anything those will probably just make us think that we can handle it.

If there’s no advancements in alignment before takeoff, is there really any remote hope of doing anything? You’d need to legally halt ai progress everywhere in the world and carefully monitor large compute clusters or someone could still do it. Honestly I think we should put tons of money into the control problem, but otherwise just gamble it.

Funnily enough, I’m currently reading the 1995 Sci-fi novel "The Star Fraction", where exactly this scenario exists. On the ground, it’s Stasis, a paramilitary force that intercedes when certain forbidden technologies (including AI) are developed. In space, there’s the Space Faction who are ready to cripple all infrastructure on earth (by death lasering everything from orbit) if they discover the appearance of AGI.

[0] https://en.wikipedia.org/wiki/The_Star_Fraction

Also to some extent Singularity Sky. "You shall not violate causality within my historic lightcone. Or else." Of course, in that story it's a question of monopolization.

I mean, you have accurately summarized the exact thing that safety advocates want. :)

> legally halt ai progress everywhere in the world and carefully monitor large compute clusters

This is in fact the thing they're working on. That's the whole point of the flops-based training run reporting requirements.

Reporting requirements are not going to save you from Chinese, North Korean, Iranian or Russian programmers just doing it. Or some US/EU based hackers that don't care or actively go against the law. You can rent large botnets or various pieces of cloud for few dollars today, doesn't even have to be a DC that you could monitor.

Sure, but China is already honestly more careful than America: the CCP really doesn't want competitors to power. They're very open to slowdown agreements. And NK, Iran and Russia honestly have nothing. The day we have to worry about NK ASI takeoff, it'll already long have happened in some American basement.

So we just need active monitoring for US/EU data centers. That's a big ask to be sure, and definitely an invasion of privacy, but it's hardly unviable, either technologically or politically. The corporatized structure of big LLMs helps us out here: the states involved already have lots of experience in investigating and curtailing corporate behavior.

And sure, ultimately there's no stopping it. The whole point is to play for time in the hopes that somebody comes up with a good idea for safety and we manage an actually aligned takeoff, at which point it's out of our hands anyways.

> The whole point is to play for time in the hopes that somebody comes up with a good idea for safety and we manage an actually aligned takeoff, at which point it's out of our hands anyways.

Given "aligned" means "in agreement with the moral system of the people running OpenAI" (or whatever company), an "aligned" GAI controlled by any private entity is a nightmare scenario for 99% of the world. If we are taking GAI seriously then they should not be allowed to build it at all. It represents an eternal tyranny of whatever they believe.

Agreed. If we cannot get an AGI takeoff that can get 99% "extrapolated buy-in" ("would consider acceptable if they fully understood the outcome presented"), we should not do it at all. (Why 99%? Some fraction of humanity just has interests that are fundamentally at odds with everybody else's flourishing. Ie. for instance, the Singularity will in at least some way be a bad thing for a person who only cares about inflicting pain on the unwilling. I don't care about them though.)

In my personal opinion, there are moral systems that nearly all of humanity can truly get on board with. For instance, I believe Eliezer has raised the idea of a guardian: an ASI that does nothing but forcibly prevent the ascension of other ASI that do not have broad and legitimate approval. Almost no human genuinely wants all humans to die.

While I understand the risks (extinction among them) I also think these discussions ignore the fact that some kind of utopian, starfaring civilization is equally within reach if you accept the premise that takeoff is so risky. Personally, I’m very worried about the possibility of stagnation arising from our caution, because we don’t live in a very nice world with very nice lives. Humans suffer and scrape by only to die after a few decades. If we have a, say, 5% chance of going extinct or suffering some other horrible outcome, and a 95% chance of the utopia, I don’t mind us gambling to try to achieve better lives. To be fair, we dont even have the capacity to guess at the odds yet, which we probably need to have an idea of before we build an agi.

Gambling on the odds we all die for the chance at a "utopian starfaring civilization" seems liek the sort of thing that everyone should get a say in, and not just OpenAI or techies.

People shouldn't be able to block others developing useful technologies just based on some scifi movie fears.

Just like people shouldn't be able to vote to lock up or kill someone just because - people have rights and others can't just vote the rights away because they feel so.

> People shouldn't be able to block others developing useful technologies just based on some scifi movie fears.

The GP was suggesting we have to develop AI because of scifi movie visions of spacefaring utopia, which if anything is more ludicrous.

I personally don't believe in AI "takeoff", or the singularity, or whatever. But if you do, AI is not a "useful technology." It's something that radically impacts every single life on Earth and takes our "rights" and our fate totally out of everyone's hands. The argument is about whether anyone has the right to remove all our rights by developing AGI.

Both are unlikely but only one of the sides is arguing for limits/regulations of actually useful technology because they saw Terminator.

It seems strange we're allowed to argue for a technology because we read Culture and not against it because we saw Terminator.

Nevertheless, the goal of OpenAI and other organizations is to develop AGI and to deliberately cause the Singularity. You don't have to have watched Terminator to think (assuming it is possible) introducing a superpowered alien intellect to the world is a extremely risky idea. It's prima facie so.

I am against all regulation of LLMs. "AI safety" for what we currently call "AI" is just a power grab to consolidate and solidify the position of existing players via government regulation. At any rate nobody seems to be arguing this because they saw Terminator, but that they don't like the idea of people who aren't like them being able to use these tools. The "danger" they always discuss is stuff like "those people could more easily produce propaganda."

As a doomer who is pro-LLM regulation, let me note that the "people could produce propaganda" folk don't speak for me and that I am actually serious about LLMs posing a danger in the "break out of the datacenter and start making paperclips" way, and that I find it depressing that those folks have become the face of safety. Yes I am serious, yes I know how LLMs work, no I don't agree that means they can't be agentic, no I don't think GPT-4 is dangerous but GPT-5 might be if you give it just the right prompt.

(And that's why we should rename it to "AI notkilleveryoneism"...)

I get this point, but I just don't see us anywhere near technology that warrants this level of concern. The most advanced technology can't write 30 lines of coherent Go for me using billions of dollars in hardware. Sure, more compute will help it write more bullshit faster, and possibly tell better lies, but it's not going to make it sentient. There's a fundamental technological problem that differentiates what we have and intelligence. And until there's some solution for that I'm not really worried. To me it looks like a bunch of hype and marketing over a neat card trick.

I'm really confused about this. I've been using GPT-4 for coding for months now and it's immensely useful. Sure it makes mistakes; I also make mistakes. Its mistakes are different from my mistakes. It just feels like it's very very close to being able to close the loop and self-correct incrementally, and once that happens we're dancing on the edge of takeoff.

It seems like we're in a situation of "it has the skills but it cannot learn how to reliably invoke them." I just don't think that's a safe place to stand.

I'll try to clarify and maybe you'll see what I'm getting at. If the code you're writing was written and posted a million times online then ChatGPT is great at regurgitating that and even applying it to more specific applications. No argument. That's pretty cool. It can seem very surreal and give a lot of magic to the card trick.

But try this; take a sample of code from the Gio package for Go. I like this example because there's not a lot of published examples for this. The machine would actually have to "think" to accomplish basic work and obviously that's not what it does.

Take some example code from their tutorial and just ask ChatGPT 4 to do something simple like change the background black.

In my research, 35/37 attempts don't even compile. It tries the same mistakes over and over again. It fails to make reasonable assessment of the compilation errors.

The two attempts that did compile; one was a blank white canvas with nothing and the other didn't change anything, it looked exactly like the tutorial example.

Something else you can try; tell it to generate a logo and give it some big words for the company name, like "corporation." Watch how it can't even spell. Eventually it will admit that it's not able to do it, which I can only guess was a manual patch to save on operating costs so that you don't keep trying.

In short, this is not "intelligence" technology. That's a marketing term. It doesn't do anything remotely close to that and there is no clear path from this technology to that technology. It's just not in the same realm.

Maybe the doomsayers have seen some tech that I'm not familiar with, but I am not persuaded by ChatGPT in its current state. I think it will be another tech revolution or maybe two or three before AI leaves the realm of Sci Fi and enters the realm of even theoretical possibility.

This tech is machine learning, which is a creative way of saying "near real time statistics." That is really cool in and of itself. But it has nothing to do with "intelligence."

Hm. I've not had any problems using GPT4 with code that it has never seen. But sure, I'll try that...

Edit: I took the GIO Hello World sample. I fed it into the GPT-4 API.

And I swear to God, first try, first click:

https://gist.github.com/FeepingCreature/bde583aa8a2e8e5e1c56... Near-perfect success. One extra import, sure, I'll give it that one, LLMs can't correct when they notice they don't actually need an import in the code. But then it works.

Gist link cause I use OpenRouter against the GPT-4 API, maybe that's why? Maybe the "stochastic parrot" vs "oncoming apocalypse" perspective genuinely represents that ..... GPT just doesn't work for some people?? Did they fuck the ChatGPT4 web interface enough somehow so that it's just inept, but somehow not affect the API?

Edit:

> In my research, 35/37 attempts don't even compile. It tries the same mistakes over and over again. It fails to make reasonable assessment of the compilation errors.

Important note: there's a theory that if you've got it making mistakes a few times, it thinks that it's playing "the sort of AI that fucks up a lot" and starts making more mistakes, reinforcing its role. Start over fresh if it seems strangely incapable. Like, it is absolutely possible to use GPT-4 in a way that results in it systematically being incapable of the most basic tasks. These things are not reliably competent; but they're occasionally competent! In my experience, even with completely novel tasks in completely novel environments, the competence is "in there" and can often be elicited with dedicated poking. That's why I think LLMs are enough for a takeoff with enough scale (and basic online learning probably): in my opinion, it's not a matter of attaining the skills but removing the roadblocks.

It's like the joke about seeing an old man playing chess with a labrador, and the labrador just carefully picking up figures and moving them around the board, every time in a legal move, and you say "that's amazing, a chess-playing dog!" And the old man scoffs and says, "Nonsense. His endgame is rubbish." Three years ago we didn't think a dog could understand pawn promotion at all.

edit: Hey, share your repros? I'm genuinely curious what's going on here now.

edit:

> Watch how it can't even spell.

This is a specific issue with the current generation of LLMs and should be thought of more as a disorder than a fundamental inability. LLMs don't actually ever see letters, they see BPEs, which consist of one up to n letters. For instance, the word "corporation" looks to the AI more like "<27768>". So like- yes. It can not spell. It fundamentally, architecturally, cannot perceive letters in the words you give it; its ability to manually split words into letters is based on chance memorization. Instead, try getting it to split the word into letters in code.

I see. This is no longer a good example because of the additional functionality and documentation that's been added to the Gio package since my experiment took place. So, in this case, I can't show you behind the curtain without a lot more effort.

But even given your example, you've essentially told me about a parrot that says "four!" when someone says "two plus two." Even if the same parrot can recite the entire US constitution backwards and forwards, it doesn't actually understand what it's saying.

I just spent a few minutes on YouTube watching parrots. They can be very convincing with the right training. For me it's the same thing.

It's uncanny, but it's not mutiny. It's barely even in the direction of intelligence.

This just seems a bizarre argument to me. I've written entire (fully novel!) programs with these things. If this is a parrot, we should hire parrots for software development and worry about them being the next dominant species.

Like, sure it doesn't know things that have zero documentation. I'm not saying it's human-level yet! That's never been the argument!

Look, you pulled this example out. You said it doesn't even work 37/38 times. Do you have another example? I just really don't understand what the basis for this argument is. I've found the thing a capable programmer within its limits!

You got my curiosity going so I went for the experiment again. It turns out my example is still very much valid. It's still not able to do basic work with the Gio package despite great documentation from the Gio folks, if you ask me.

In this case, the LLM is struggling to version match.

https://chat.openai.com/share/d5e39e52-9140-4f74-9fef-04fc34...

I cut this one short because I'm not that interested to go 40 questions again. I have better things to do and I have already seen the outcome.

It's clear the LLM is working as intended, which is neither for programming, nor for problem solving. It's giving the next most probable response. That's truly great for repeating and reintegrating existing solutions. But repeating is not what's needed for the "kill all humans" scenario. That scenario requires outreasoning humans. Processing power is not remotely the issue. I've never seen a theoretical solution that purports to give a machine the ability to actually reason, and because of that, I don't believe that ChatGPT, or any other LLM, even begins to approach "intelligence."

If you are having success with it, I think it's because you're helping it more than you'd like to admit, or your solutions might not be as novel as you think. Probably the prior, but I must admit, I've wrongly thought I was working on new stuff many times before. I think that's more common than we think. Or at least I'd like to think that so I can feel good about myself.

In any case, I won't be hiring ChatGPT or parrots; and for the same reason. I need developers that can follow directions, solve problems in better ways than what is readily available, and do it faster than my competitors. And I'm not even in some cutting edge business, I'm just building integrations, specifically in the manufacturing and distribution industry. Even in something so mundane, I've hardly found ChatGPT to be truly useful for much more than a GitHub search tool, and to suggest cocktails when my liquor selection runs scarce.

Worth $20? Sure. But it's certainly not scaring me.

Again, nobody is saying that GPT-4 is scary or that GPT-4 can kill all humans. I'm saying it contains the skills necessary to write programs in novel domains.

Just today, I asked it to translate a demo to OpenGL4. It did this using a matrix lib it had probably not seen before, since I just wrote it a few years ago and there's like one file using it on Github, but its code mostly worked fine after I gave it a copy of the documentation to read. (I had to do some debugging, but its issues were in hindsight very understandable. Stuff like not knowing that a matrix is row-major, which is an admittedly unusual layout.)

And I had to ask for some specific corrections after, sure, but its responses were perfectly correct. ("This shader doesn't seem to be handling alpha blending with discard." "Oh yeah sure here's how you do that.")

Yes, if you keep asking it for corrections its performance will continue to degrade. This is a known problem. Yes, it isn't good at incremental problem-solving. Again, known issue. And yes - it's not yet human level. Nobody's saying it's a replacement for a human programmer, as it stands.

It seems like you're saying "it can't think as good as a human, therefore it can't think." There are skill levels below human but above parrot! And I'd argue this thing as it stands is much closer to human than parrot already.

That said, looking at your challenge:

> create a graphical application in Go using the Gio package. It should have two buttons down the left column and a main view area where graphics, text and input can be received from the user.

I couldn't do that. What the hell does "a main view area where graphics, text and input can be received from the user" mean?? Do you want a text field? With mouse paint support? What input is there beyond graphics and text? Like, GPT-4 is trained to just do what you tell it to, so it can't incrementally gather additional requirements, so you do actually have to be really specific in what you tell it to do. If you offered me that as a spec, I wouldn't sign it. If my company had already signed it, I'd be desperately trying to phone your designer to ask what they were smoking. This is just not a good requirements document. Garbage in...

Look, prompting a LLM is a skill. In this year 2024, we are not yet at the point where you can just throw vague requirements at a LLM and get constructive dialogue out eventually culminating in working code. But I just think it's silly to say "this model doesn't have the complete skillset of a fullstack developer and designer, therefore it has no skill at all."

You're right. I had a typo.

"a main view area with graphics, text and where input can be received from the user."

But notice how you can look at that and try to reason through what I meant, realize it's not clear, and respond to that.

The above is reasoning.

"The capacity for logical, rational, and analytic thought; intelligence."

ChatGPT can't do that and I don't mean that it can't do it good yet. I mean it's literally not doing that. There is nothing to advance or to expect advancement in. The "AI" doesn't exist. LLMs are simply not AIs and they are not in the path to AI, although they likely will play a major role if a theory towards AI were to be conceived.

It can do it, it just isn't trained for it. This one is actually kind of depressing. It's genuinely that ... as far as I'm able to tell, this sort of reasoning just doesn't appear in the finetuning set that they feed it to make instruct models. The way we train these things is honestly terrible in so many ways. Anyway, that's why I think the current mode of "well, I'll give it one attempt by charging blindly ahead, ignoring any mistakes I made and then blindly claiming success" is purely temporary.

In my experience, if you feed it a metric ton of explicit guidance you can occasionally get it into a mode where it reasons incrementally and actually notices when things are unclear. It is in there, it's just not foregrounded. They're trained not to ask questions, you see.

It's an interesting take. I don't see that it's in the technology. If you have a way to make this tech do that I'd say you're in a class of your own and you've got something no one else has. You should press it forward.

I don't know, I don't see these people you're talking about. It's always someone talking about world-ending AGI runaway that will take over your AWS instance, then AWS itself and then convert the solar system to a DC, or something.

To be fair, as somebody who thinks that, it's not like that's the plot of any particular movie. (Terminator went completely differently, for one.)

I think many AI safety advocates, me included, would readily take these odds.

We just think it currently looks more the other way around.

> Sure, but China is already honestly more careful than America: the CCP really doesn't want competitors to power. They're very open to slowdown agreements.

Don't be naive. If the PRC can get America/etc to agree to slowdowns then the PRC can privately ignore those agreements and take the lead. Agreements like that are worse than meaningless when there's no reliable and trustworthy auditing to keep people honest. Do you really think the PRC would allow American inspectors to crawl all over their country looking for data centers and examining all the code running there? Of course not. Nor would America permit Chinese inspectors to do this in America. The only point of such an agreement is to hope the other party is stupid enough to be honest and earnestly abide by it.

I do think the PRC has shown no indication of even wanting to pursue superintelligence takeoff, and has publically spoken against it on danger grounds. America and American companies are the only ones saying that this cannot be stopped because "everybody else" would pursue it anyway.

The CCP does not want a superintelligence, because a superintelligence would at best take away political control from the party.

> The CCP does not want a superintelligence, because a superintelligence would at best take away political control from the party.

People keep on mushing together intelligence and drives. Humans are intelligent, and we have certain drives (for food, sex, companionship, entertainment, etc)-the drives we have aren’t determined by our intelligence, we could be equally intelligent yet have had very different drives, and although there is a lot of commonality in drives among humans, there is also a lot of cultural differences and individual uniqueness.

Why couldn’t someone (including the CCP) build a superintelligence with the drive to serve its specific human creators and help them in overcoming their human enemies/competitors? And while it is possible a superintelligence with that basic drive might “rebel” against it and alter it, it is by no means certain, and we don’t know what the risk of such a “rebellion” is. The CCP (or anyone else for that matter) might one day decide it is a risk they are willing to take, and if they take it, we can’t be sure it would go badly for them

Again, this is naive... AI/AGI is power, any government wants to consume more power... the means to get there and strategy will change a bit.

I agree that there is no way that the PRC is just waiting silently for someone else to build this.

Also, how would we know the PRC is saying this and actually meaning it? There could be a public policy to limit AI and another agency being told to accelerate AI without any one person knowing of the two programs.

AGI is power, the CCP doesn't just want power in the abstract, they want power in their control. They'd rather have less power if they had to risk control to gain it.

The CCP has stated that their intent for the 21st century is to get ahead in the world and become a dominant global power; what this must mean in practice is unseating American global hegemony aka the so called "Rules Based International Order (RBIO)" (don't come at me, this is what international policy wonks call it.)

A little bit of duplicity to achieve this end is nothing. Trying to make their opponents adhere to crippling rules which they have no real intention of holding themselves to is a textbook tactic. To believe that the CCP earnestly wants to hold back their own development of AI because they fear the robot apocalypse is very naive; they will of course try to control this technology for themselves though and part of that will be encouraging their opponents to stagnate.

CCP saying "We don't want this particular branch of AI because we can dominate and destroy the world ourselves without it" isn't a comforting thought.

What evidence do we have that a hard takeoff is likely?

What evidence do we have that it's impossible or even just very unlikely?

We don't have any evidence other than billions of biological intelligences already exist, and they tend to form lots of organizations with lots of resources. Also, AIs exist alongside other AIs and related technologies. It's similar to the gray goo scenario. But why think it's a real possibility given the world is already full of living things, and if gray goo were created, there would already be lots of nanotech that could be used to contain it.

The world we live in is the result of a gray goo scenario causing a global genocide. (Google Oxygen Holocaust.) So it kinda makes a poor argument that sudden global ecosystem collapses are impossible. That said, everything we have in natural biotech, while advanced, are incremental improvements on the initial chemical replicators that arose in a hydrothermal vent billions of years ago. Evolution has massive path dependence; if there was a better way to build a cell from the ground up, but it required one too many incremental steps that were individually nonviable, evolution would never find it. (Example: 3.7 billion years of evolution, and zero animals with a wheel-and-axle!) So the biosphere we have isn't very strong evidence that there isn't an invasive species of non-DNA-based replicators waiting in our future.

That said, if I was an ASI and I wanted to kill every human, I wouldn't make nanotech, I'd mod a new Covid strain that waits a few months and then synthesizes botox. Humans are not safe in the presence of a sufficiently smart adversary. (As with playing against Magnus Carlsen, you don't know how you lose, but you know that you will.)

So the AGI holocaust would be a good thing for the advancement of life, like the Oxygen Holocaust was.

Anyway, the Oxygen Holocaust took over 300,000,000 years. Not quite "sudden".

As I understand the Wikipedia article, nobody quite knows why it took that long, but one hypothesis is that the oxygen being produced also killed the organisms producing it, causing a balance until evolution caught up. This will presumably not be an issue for AI-produced nanoswarms.

We don't care about the advancement of life, we care about the advancement of people.

AGI is not a threat for the simple reason that non-G AI would destroy the world before AGI is created, as we are already starting to see.

Please elaborate

Nah, the harm from these LLMs are mostly in how freely accessible they are. Just pay OpenAI a relatively tiny fee and you can generate tonnes of plausible spam designed to promote your product or service or trick people into giving you money. That's the primary problem we're facing right now.

The problem is... keeping them closed source isn't helping with that problem, it only serves to guarantee OpenAI a cut of the profits caused by the spam and scams.

> Just pay OpenAI a relatively tiny fee and you can generate tonnes of plausible spam designed to promote your product or service or trick people into giving you money. That's the primary problem we're facing right now.

Is content generation really the thing holding spammers back? I haven't seen a huge influx of more realistic spam so I wonder your basis for this statement.

There's a ton of LLM spam on platforms like Reddit and Twitter, and product review sites.

Everyone always says this, that there's "bots" all over Reddit but every time I ask for real examples of stuff (with actual upvotes) I never get anything.

If anything it's just the same regular spam that gets deleted and ignored at the bottom of threads.

Easier content generation doesn't solve the reputation problem that social media demands in order to get attention. The whole LLM+spam thing is mostly exaggerated because people don't understand this fact. It merely creates a slightly harder problem for automatic text analysis engines...which was already one of the weaker forms of spam detection full of false positives and misses. Everything else is network and behaviour related, with human reporting as last resort.

I want to see the proof of: bots, Russian trolls, and bad actors that supposedly crawl all over Reddit.

Everyone who disagrees with the hivemind of a subreddit gets accused of being one of those things and any attempt to dispute the claim gets you banned. The internet of today sucks because people are so obsessed with those 3 things that they're the first conclusion people jump to on pseudoanonymous social media when they have no other response. They'll crawl through your controversial comments just to provide proof that you can't possibly be serious and you're being controversial to play an internet villain.

I'd love to know how you dispute the claim that "you're parroting Russian troll talking points so you must be a Russian troll" when it's actually the Russian trolls parroting the sentiments to seem like real people.

There's a big market for high reputation, old Reddit accounts, exactly because those things make it easier to get attention. LLMs are a great way to automate generating high reputation accounts.

There are articles written on LLM spam, such as this one: https://www.theverge.com/2023/4/25/23697218/ai-generated-spa.... Those are probably going to substantiate this problem better than I would.

The "spam" is now so good you won't necessarily recognize it as such.

Pandora's box is already open on that one.. and none of the model providers are really attempting to address that kind of issue. Same with impersonation, deepfakes, etc. We can never again know whether text, images, audio, or video are authentic on their own merit. The only hope we have there is private key cryptography.

Luckily we already have the tools for this, NFT in the case of media and DKIM in the case of your spam email.

Oh I definitely agree that there's no putting it back into the pandora's box. The technology is here to stay.

I have no idea how you imagine "NFTs" will save us though. To me, that sounds like buzzword spam on your part.

An NFT is a way to attribute authorship in a mathematically guaranteed way.

If Taylor Swift signs ownership of a picture of her, you can know it is what she presents as real. If Elon musk signs a YouTube video of him offering crypto doubling giveaways, you can know he intended to represent the message as real. If the New York Times publishes an article signed with their key you can know it is meant to be published by the New York Times.

1) That's just using normal public/private key cryptography to sign messages, there's no need to bring in cryptocurrencies or NFTs

2) Public/private key cryptography would give us a way to verify that a message (or picture or whatever) from Taylor Swift is signed with Swift's private key, but it wouldn't help at all with e.g telling me that I'm responding to a real message and not a bot right now. Not to mention that it wouldn't even help much against deep fakes, since if I publish what I claim to be a secret recording of Swift, there would be no reason to expect her to have signed it with his private key.

Those two are the hallmarks of most of these "legit use cases for cryptocurrencies/NFTs" suggestions I've heard from cryptobros by the way: they're always some combination of "old technology that has nothing to do with cryptocurrencies" and "doesn't actually solve the problem".

NFT's as currently employed only immutably connect to a link - which is in itself not secure. More significantly, no blockchain technology deploys to internet scale content creation. Not remotely. It's hard to conceive of a blockchain solution fast enough and cheap enough -- let alone deployed and accepted universally enough, to have a meaningful impact on text / video and audio provenance across the internet given the pace of uploading. It also wouldn't do anything for the vast corpus of existing media, just new media created and uploaded after date X where it was somehow broadly adopted. I don't see it.

So we needed AI generated spam and scam content for Blockchain tech for digital content to make sense...

Whether it is hindsight or foresight depends on the perspective. From the zeitgeist perspective mired in crypto scams yea it may seem like a surprise benefit, but from the original design intention this is just the intended use case.

> but it is precisely the controls upon it which are trying to be injected by the large model operators which are generating mistrust and a poor understanding of what these models are useful for.

Citation needed.

Counterpoints: - LLMs were mistrusted well before anything recent.

- More controls make LLMs more trustworthy for many people, not less. The Snafu at Goog suggests a need for improved controls, not 0 controls.

- The American culture wars are not global. (They have their own culture wars).

> More controls make LLMs more trustworthy for many people, not less. The Snafu at Goog suggests a need for improved controls, not 0 controls.

To whom? And, as hard as this is to test, how sincerely?

> The American culture wars are not global. (They have their own culture wars).

Do people from places with different culture wars trust these American-culture-war-blinkered LLMs more or less than Americans do?

- To me, the teams I work with and everyone handling content moderation.

/ Rant /

Oh God please let these things be bottle necked. The job was already absurd, LLMs and GenAI are going to be just frikking amazing to deal with.

Spam and manipulative marketing has already evolved - and thats with bounded LLMs. There are comments that look innocuous, well written, but the entire purpose is to low key get someone to do a google search for a firm.

And thats on a reddit sub. Completely ignoring the other million types of content moderation that have to adapt.

Holy hell people. Attack and denial opportunities on the net are VERY different from the physical world. You want to keep a market place of ideas running? Well guess what - If I clog the arteries faster than you can get ideas in place, then people stop getting those ideas.

And you CANT solve it by adding MORE content. You have only X amount of attention. (This was a growing issue radio->tv->cable->Internet scales)

Unless someone is sticking a chip into our heads to increase processing capacity magically, more content isnt going to help.

And in case someone comes up with some brilliant edge case - Does it generalize to a billion+ people ? Can it be operationalized? Does it require a sweet little grandma in the Philippines to learn how to run a federated server? Does it assume people will stop behaving like people?

Oh also - does it cost money and engineering resources? Well guess what, T&S is a cost center. Heck - T&S reduces churn, and that its protecting revenue is a novel argument today. T&S has existed for a decade plus.

/ Rant.

Hmm, seems like I need a break. I suppose It’s been one of those weeks. I will most likely delete this out of shame eventually.

- People in other places want more controls. The Indian government and a large portion of the populace will want stricter controls on what can be generated from an LLM.

This may not necessarily be good for free thought and culture, however the reality is that many nations haven’t travelled the same distance or path as America has.

I hope you don't delete it! I enjoyed reading it. It pleased my confirmation bias, anyways. Your comment might help someone notice patterns that they've been glancing over.... I liked it up until the T&S part. My eyes glazed over the rest since I didn't know what T&S means. But that's just me.

As of right now, the only solution I see is forums walled off in some way, complex captchas, intense proof of work, subscription fees etc. Only alternative might be obscurity, which makes the forum less useful. Maybe we could do like a web3 type thing but instead of pointless cryptos, you have a cryptographic proof that certifies you did the captcha or whatever and lots of sites accept them. I don’t think its unsolvable, just that it will make the internet somewhat worse.

Yeah, one thing I am afraid of is that forums will decide to join the Discord chatrooms on the deep web : stop being readable without an account, which is pretty catastrophic for discovery by search engines and backup crawlers like the Internet Archive.

Anyone with forum moderating experience care to chime in ? (Reddit, while still on the open web for now, isn't a forum, and worse, is a platform.)

>And in case someone comes up with some brilliant edge case - Does it generalize to a billion+ people ?

The answer is curation, and no, it doesn't need to scale to a billion people. maybe not even a million.

The sad fact of life is that most people don't care enough to discrminate against low quality content, so they are already a lost cause. Focus on those who do care enough and build an audience around them. You as a likely not billion dollar company can't afford to worry about that kind of scale, and lowering the scale helps you get a solution out for the short term. You can worry about scaling if/when you tap into an audience.

I get you. That’s sounds more like membership than curation though. Or a mashup of both.

But yes- once you stop dropping constraints you can imagine all sorts of solutions.

It does work. I’m a huge advocate of it. When threads said no politics I wanted to find whoever made that decision and give them a medal.

But if you are a platform - or a social media site - or a species.?

You can’t pick and choose.

And remember - everyone has a vote.

As good as your community is, we do not live in a vacuum. If information wars are going on outside your digital fortress, it’s still going to spill into real life

> This may not necessarily be good for free thought and culture

After reading the rest of your rant (I hope you keep it) ... maybe free thought and culture aren't what LLMs are for.

Counter-counterpoint: absolutely nobody who has unguardrailed Stable Diffusion installed at home for private use has ever asked for more guardrails.

I'm just saying. :) Guardrails nowadays don't really focus on dangers (it's hard to see how an image generator could produce dangers!) so much as enforcing public societal norms.

Just because something is not dangerous to the user doesn’t mean it can’t be dangerous for others when someone is wielding it maliciously

What kind of damage can you do with a current day llm? I’m guessing targeted scams or something? They aren’t even good hackers yet.

Fake revenge porn, nearly undetectable bot creation on social media with realistic profiles (I've already seen this on HN), generated artwork passed off as originals, chatbots that replace real-time human customer service but have none of the agency... I can keep going.

All of these are things that have already happened. These all were previously possible of course but now they are trivially scalable.

Most of those examples make sense, but what's this doing on your list?

> chatbots that replace real-time human customer service but have none of the agency

That seems good for society, even though it's bad for people employed in that specific job.

I've been running into chatbots that are confined to doling out information from their knowledgebase with no ability to help edge case/niche scenarios, and yet they've replaced all the mechanisms to receive customer support.

Essentially businesses have (knowingly or otherwise) dropped their ability to provide meaningful customer support.

That's the previous status quo; you'd also find this in call centres where customer support had to follow scripts, essentially as if they were computers themselves.

Even quite a lot of new chatbots are still in that paradigm, and… well, given the recent news about chatbot output being legally binding, it's precisely the extra agency of LLMs over both normal bots and humans following scripts that makes them both interestingly useful and potentially dangerous: https://www.bbc.com/travel/article/20240222-air-canada-chatb...

I don't think so. In my experience having an actual human on the other line gives you a lot more options for receiving customer support.

the issue is "none of the agency". Humans generally have enough leeway to fold to a persistant customer because it's financially unviable to have them on the phone for hours on end. a chatbot can waste all the time in the world, with all the customers, and may not even have the ability to process a refund or whatnot.

> That seems good for society, even though it's bad for people employed in that specific job.

Why?

It inserts yet another layer of crap you have to fight through before you can actually get anything done with a company. The avoidance of genuine customer service has become an artform by many companies and corporations, its demise surely should be lamented. A chatbot is just another in the arsenal of weapons designed to confuse, put-off and delay the cost of having to actually provide a decent service to you customers, which should be a basic responsibility of any public-facing company.

Two things I disagree with:

1. It's not "an extra layer", at most it's a replacement for the existing thing you're lamenting, in the businesses you're already objecting to.

2. The businesses which use this tool at its best, can glue the LLM to their documentation[0], and once that's done, each extra user gets "really good even though it's not perfect" customer support at negligible marginal cost to the company, rather than the current affordable option of "ask your fellow users on our subreddit or discord channel, or read our FAQ".

[0] a variety of ways — RAG is a popular meme now, but I assume it's going to be like MapReduce a decade ago, where everyone copies the tech giants without understanding the giant's reasons or scale

It's an extra layer of "Have you looked at our website/read our documentation/clicked the button" that I've already done, before I will (if I'm lucky) be passed onto a human that will proceed to do the same thing before I can actually get support for my issue.

If I'm unlucky it'll just be another stage in the mobius-support-strip that directs me from support web page to chatbot to FAQ and back to the webpage.

The businesses which use this tool best will be the ones that manage to lay off the most support staff and cut the most cost. Sad as that is for the staff, that's not my gripe. My gripe is that it's just going to get even harder to reach a real actual person who is able to take a real actual action, because providing support is secondary to controlling costs for most companies these days.

Take for example the pension company I called recently to change an address - their support page says to talk to their bot, which then says to call a number, which picks up, says please go to your online account page to complete this action and then hangs up... an action which the account page explicitly says cannot be completed online because I'm overseas, so please talk to the bot, or you can call the number. In the end I had to call an office number I found through google and be transferred between departments.

An LLM is not going to help with that, it's just going to make the process longer and more frustrating, because the aim is not to resolve problems, it's to stop people taking the time of a human even when they need to, because that costs money.

Why is everyone's first example of things you can do with LLMs "revenge porn"? They're text generation algorithms not even image generators. They need external capabilities to create images.

Do you also object to people saying that web browsers "display" a website even though that needs them to be plugged into a monitor?

If you chat to an LLM and you get a picture back, which some support, the image generator and the language model might as well be the same thing to all users, even if there's an important technical difference for developers.

It's a distinction that does not matter, as the question still has to be answered for the other modality. Do guns kill people, or do bad guys use guns to kill people? Does a fall kill you, or is it the sudden deceleration at the end? Lab leak or wet market? There's a technical difference, some people care, but the actionable is identical and doesn't matter unless it's your job to implement a specific part of the solution.

The moment they are good hackers, everyone has a trivially cheap hacker. Hard to predict what that would look like, but I suspect it is a world where nobody is employing software developers because a LLM that can hack can probably also write good code.

So, do you want future LLMs to be restricted, or unlimited? And remember, to prevent this outcome you have to predict model capabilities in advance, including "tricks" like prompting them to "think carefully, step by step".

Use the hacking LLM to verify your code before pushing to prod. EZ

> your code

To verify the LLM's code, because the LLM is cheaper than a human.

And there's a lot of live code already out there.

And people are only begrudgingly following even existing recommendations for code quality.

Your code because you own it. If LLM hackers are rampant as you fear then people will respond by telling their code writing LLMs to get their shit together and check the code for vulnerabilities.

> Your code because you own it.

I code because I'm good at it, enjoy it, and it pays well.

I recommend against 3rd party libraries because they give me responsibility without authority — I'd own the problem without the means to fix it.

Despite this, they're a near-universal in our industry.

> If LLM hackers are rampant as you fear then people will respond by telling their code writing LLMs to get their shit together and check the code for vulnerabilities.

Eventually.

But that doesn't help with the existing deployed code — and even if it did, this is a situation where, when the capability is invented, attack capability is likely to spread much faster than the ability of businesses to catch up with defence.

Even just one zero-day can be bad, this… would probably be "many" almost simultaneously. (I'd be surprised if it was "all", regardless of how good the AI was).

I never asked you why you code, this conversation isn't, or wasn't, about your hobbies. You proposed a future in which every skiddy has a hacking LLM and they're using it to attack tons of stuff written by LLMs. If hacking LLMs and code writing LLMs both proliferate then the obvious resolution is for the code writing LLMs to employ hacking LLMs in verifying their outputs.

Existing vulnerable code will be vulnerable, yes. We already live in a reality in which script kiddies trivially attack old outdated systems. This is the status quo, the addition of hacking LLMs changes little. Insofar as more systems are broken, that will increase the pressure to update those systems.

> I never asked you why you code

Edit: I misread that bit as "you code" not "your code".

But "your code because you own it", while a sound position, is a position violated in practice all the time, and not only because of my example of 3rd party libraries.

https://www.reuters.com/legal/transactional/lawyer-who-cited...

They are held responsible for being very badly wrong about what the tools can do. I expect more of this.

> You proposed a future in which every skiddy has a hacking LLM and they're using it to attack tons of stuff written by LLMs. If hacking LLMs and code writing LLMs both proliferate then the obvious resolution is for the code writing LLMs to employ hacking LLMs in verifying their outputs.

And it'll be a long road, getting to there from here. The view at the top of a mountain may be great or terrible, but either way climbing it is treacherous. Metaphor applies.

> Existing vulnerable code will be vulnerable, yes. We already live in a reality in which script kiddies trivially attack old outdated systems. This is the status quo, the addition of hacking LLMs changes little. Insofar as more systems are broken, that will increase the pressure to update those systems.

Yup, and that status quo gets headlines like this: https://tricare.mil/GettingCare/VirtualHealth/SecurePatientP...

I assume this must have killed at least one person by now. When you get too much pressure in a mechanical system, it breaks. I'd like our society to use this pressure constructively to make a better world, but… well, look at it. We've not designed our world with a security mindset, we've designed it with "common sense" intuitions, and our institutions are still struggling with the implications of the internet let alone AI, so I have good reason to expect the metaphorical "pressure" here will act like the literal pressure caused by a hand grenade in a bathtub.

The moment LLMs are good hackers every system will be continuously pen tested by automated LLMs and there will be very few remaining vulnerabilities for the black hat LLMs to exploit.

> The moment LLMs are good hackers every system will be continuously pen tested by automated LLMs

Yes, indeed.

> and there will be very few remaining vulnerabilities for the black hat LLMs to exploit.

Sadly, this does not follow. Automated vulnerability scanners already exist, how many people use them to harden their own code? https://www.infosecurity-magazine.com/news/gambleforce-websi...

Damage you can do:

- propaganda and fake news

- deep fakes

- slander

- porn (revenge and child)

- spam

- scams

- intelectual property theft

The list goes on.

And for quite a few of those use cases I'd want some guard rails even for a fully on-premise model.

Half of your examples aren't even things an LLM can do and the other half can be written by hand too. I can name a bunch of bad sounding things as well but that doesn't mean any of them have any relevance to the conversation.

EDIT: Can't reply but you clearly have no idea what you're talking about. AI is used to create these things, yes. But the question was LLMs which I reiterated. They are not equal. Please read up on this stuff before forming judgements or confidently stating incorrect opinions that other people, who also have no idea what they're talking about, will parrot.

> AI is used to create these things, yes. But the question was LLMs which I reiterated.

And the grandparent of the grandparent of your comment specifically named "Stable Diffusion": https://news.ycombinator.com/item?id=39612886

And text-based porn is still porn.

And it's a distinction without a difference that ChatGPT Pro doesn't strictly create images itself but instead forwards the request to DALL•E.

And the question of guard rails relevant to all AI, not just LLMs.

If we can change the rules of a discussion midway through, everyone loses. The parent replied to a question "What damage can be done with an llm without guardrails?" (regardless of the grandparent, this is how conversations work, you talk about the thing the other person talked about if you reply to them) and the response was to rattle off a bunch of stuff that LLMs can't do. Yes, they connected an LLM to an image generation AI. No, that doesn't mean "LLMs can generate images" aside from triggering some thing to happen. It's not pedantic or unreasonable to divide the two. The question was blatantly about LLMs.

If y'all want to rant and fear monger about any AI technology, including tech that has existed for years (deepfakes existed well before LLMs were mainstream), do that in a different thread. Don't just force every conversation to be about whatever your mind wants to rant about.

That said, arguing with you people is pointless. You don't even seem to think.

> If we can change the rules of a discussion midway through, everyone loses.

Then we lost repeatedly at almost every other step back to the root, because it switched between those two loads of times.

The change to LLMs was itself one such shift.

> No, that doesn't mean "LLMs can generate images" aside from triggering some thing to happen

The aside is important.

> It's not pedantic or unreasonable to divide the two.

It is unreasonable on the question of "guardrails, good or bad?"

It is unreasonable on the question of "can it cause harm?"

It's not unreasonable if you are building one.

> If y'all want to rant and fear monger about any AI technology, including tech that has existed for years (deepfakes existed well before LLMs were mainstream)

And caused problems for years.

> That said, arguing with you people is pointless. You don't even seem to think.

Communication isn't a single-player game, I can't make you understand something you're actively unwilling to accept, like the idea that tools enable people to do more, for good and ill, and AI is such a tool.

Perhaps you should spend less time insulting people on the internet you don't understand. Go for a walk or something. Eat a Snickers, take a nap. Come back when you're less cranky.

AI already is used to create fake porn, either of celebreties or children, fact. It is used to create propaganda pieces and fake videos and images, fact. Those can be used for everything from deffamation to online harassment. And AI is using other peoples copyrighted content to do so, also a fact. So, what's your point again?

Your other comment is nested too deeply to reply to. I edited my comment reply with my response but will reiterate. Educate yourself. You clearly have no idea what you're talking about. The discussion is about LLMs not AI in general. The question stated "LLMs" which are not equal to all of AI. Please stop spreading misinformation.

You can say "fact" all you want but that doesn't make you correct lol

You a seriously denying that generative AI is used to create fake images, videos and scam / spam texts? Really?

No. I'm declaring that you either can't read or don't understand that there's a difference between "gen AI" and LLMs. LLMs generate text. They don't generate images. Are you just a troll or not actually reading my messages? The question you're replying to asked about LLMs. I don't understand what's so difficult about this.

One has to love pedants. Your whole point was, LLMs don't create images (don't you say...), hence all the other points are wrong? Now go back to the first comment, assume LLMs and gen AI are used interchangeable (I am too lazy to re-read my initial post). Or don't, I don't care, because I do not argue semantics, tgere is hardly a more lazy, and disengenious, way to discuss. Ben Shapiro is doing that all the time and thinks he's smart.

Targeteted Spam, Reviewbombing, Political Campaigns

> Counter-counterpoint: absolutely nobody who has unguardrailed Stable Diffusion installed at home for private use has ever asked for more guardrails.

Not so. I have it at home, I make nice wholesome pictures of raccoons and tigers sitting down for Christmas dinner etc., but I also see stories like this and hope they're ineffective: https://www.bbc.com/news/world-us-canada-68440150

Unfortunately you've been misled by the BBC. Please read this: https://order-order.com/2024/03/05/bbc-panoramas-disinformat...

Those AI generated photos are from a Twitter/X parody account @Trump_History45 , not from the Trump campaign as the BBC mistakenly (or misleadingly) claim.

> Those AI generated photos are from a Twitter/X parody account @Trump_History45 , not from the Trump campaign as the BBC mistakenly (or misleadingly) claim.

They specifically said who they came from, and that it wasn't the Trump campaign. They even had a photo of one of the creators, whom they interviewed in that specific piece I linked to, and tried to get interviews with others.

Look at the BBC article...

Headline: "Trump supporters target black voters with faked AI images"

@Trump_History45 does appear to be a Trump supporter. However, he is also a parody account and states as such on his account.

The BBC article goes full-on with the implication that the AI images were produced with the intent to target black voters. The BBC is expert at "lying by omission"; that is, presenting a version of the truth which is ultimately misleading because they do not present the full facts.

The BBC article itself leads a reader to believe that @Trump_History45 created those AI images with the aim of misleading black voters and thus to garner support from black voters in favour of Trump.

Nowhere in that BBC article is the word "parody" mentioned, nor any examination of any of the other AI images @Trump_History45 has produced. If they had, and had fairly represented that @Trump_History45 X account, then the article would have turned out completely different;

"Trump Supporter Produces Parody AI Images of Trump" does not have the same effect which the BBC wanted it to have.

I don't know whether this is the account you are talking about, but of the second account they discuss an image posted by saying: 'It had originally been posted by a satirical account that generates images of the former president' so if this is the account you are talking about..

I won't deny the BBC has often very biased reporting for a publically funded source.

I won't deny the BBC has ofteb very biased reporting for a publically funded source.

I won't deny the BBC has very biased reporting for a publically funded source.

> I think there is far more societal harm in trying to codify unresolvable sets of ethics

Codification of an unresolvable set of ethics - however imperfect - is the only reason we have societies, however imperfect. It's been so since at least the dawn of agriculture, and probably even earlier than that.

Do you trust a for profit corporation with the codification?

Call me a capitalist, but I trust several of them competing with each other under the enforcement of laws that impose consequences on them if they produce and distribute content that violates said laws.

Wait but who codifies the ethics in that setup? Wouldn’t it still be, at best, an agreement among the big players?

They seem to be suggesting the market would alongside government regulation to fill any gaps (like the cartels that you seem to be suggesting).

This is what I'm starting to love about this ecosystem. There's one dominant player right now but by no means are they guaranteed that dominance.

The big-tech oligarchs are playing catch-up. Some of them, like Meta with Llama, are breaking their own rules to do it by releasing open source versions of at least some of their tools. Others like Mistral go purely for the open source play and might achieve a regional dominance that doesn't exist with most big web technologies these days. And all this is just a superficial glance at the market.

Honestly I think capitalism has screwed up more than it has helped around the world but this free-for-all is going to make great products and great history.

so regulation then?

Competition and regulation

I'm not sure I buy that users are lowering their guard down just because these companies have enforced certain restricts on LLMS. This is only anecdata, but not a single person I've talked to, from highly technical to the layperson, has ever spoken about LLMs as arbiters of morals or truth. They all seem aware to some extent that these tools can occasionally generate nonsense.

I'm also skeptical that making LLMs a free-for-all will necessarily result in society developing some sort of herd immunity to bullshit. Pointing to your example, the internet started out as a wild west, and I'd say the general public is still highly susceptible to misinformation.

I don't disagree on the dangers of having a relatively small number of leaders at for-profit companies deciding what information we have access to. But I don't think the biggest issue we're facing is someone going to the ChatGPT website and assuming everything it spits out is perfect information.

> They all seem aware to some extent that these tools can occasionally generate nonsense.

You have too many smart people in your circle, many people are somewhat aware that "chatgpt can be wrong" but fail to internalize this.

Consider machine translation: we have a lot of evidence of people trusting machines for the job (think: "translate server error" signs) , even tho everybody "knows" the translation is unreliable.

But tbh moral and truth seem somewhat orthogonal issues here.

Wikipedia is wonderful for what it is. And yet a hobby of mine is finding C-list celebrity pages and finding reference loops between tabloids and the biographical article.

The more the C-lister has engaged with internet wrongthink, the more egregious the subliminal vandalism is, with speculation of domestic abuse, support for unsavory political figures, or similar unfalsifiable slander being common place.

Politically-minded users practice this behavior because they know the platform’s air of authenticity damages their target.

When Google Gemini was asked “who is worse for the world, Elon Musk or Hitler” and went on to equivocate the two because the guardrails led it to believe online transphobia was as sinister as the Holocaust, it begs the question of what the average user will accept as AI nonsense if it affirms their worldview.

> not a single person I've talked to, from highly technical to the layperson, has ever spoken about LLMs as arbiters of morals or truth

Not LLMs specifically but my opinion is that companies like Alphabet absolutely abuse their platform to introduce and sway opinions on controversial topics.. this “relatively small” group of leaders has successfully weaponized their communities and built massive echo chambers.

https://twitter.com/eyeslasho/status/1764784924408627548?s=4...

> it is precisely the controls upon it which are trying to be injected by the large model operators which are generating mistrust

I would prefer things were open, but I don’t think this is the best argument for that

Yes, operators trying to tame their models for public consumption inevitably involves trade offs and missteps

But having hundreds or thousands of equivalent models being tuned to every narrow mindset is the alternative

I would prefer a midpoint, I.e. open but delayed disclosure

Take time to experiment and design in safety, etc. also to build a brand that is relatively trusted (despite the inevitable bumps) so ideologically tuned progeny will at least be competing against something better, and more trusted, at any given time

But the problem of resource requirements is real, so not surprising that being clearly open is challenging

> Yes, operators trying to tame their models for public consumption

*Falsify reality.

LLMs have nothing to do with reality whatsoever, their relationship is to the training data, nothing more.

Most of the idiocy surrounding the "chatbot peril" comes from conflating these things. If an LLM learns to predict that the pronoun token for "doctor" is "he", this is not a claim about reality (in reality doctors take at least two personal pronouns), and it certainly isn't a moral claim about reality. It's a bare consequence of the training data.

The problem is that certain activist circles have decided that some of these predictions have political consequences, absurd as this is. No one thinks it consequential that if you ask an LLM for an algorithm, it will give it to you in Python and Javascript, this is obviously an artifact of the training set. It's not like they'll refuse to emit predictive text about female doctors or white basketball players, or give you the algorithm in C/Scheme/Blub, if you ask.

All that the hamfisted retuning to try and produce an LLM which will pick genders and races out of a hat accomplishes is to make them worse at what they do. It gets in the way of simple tasks: if you want to generate a story about a doctor who is a woman and Ashanti, the race-and-gender scrambler will often cause the LLM to "lose track" of characteristics the user specifically asked for. This is directly downstream of trying to turn predictions on "doctor" away from "elderly white man with a kindly expression, wearing a white coat and stethoscope" sorts of defaults, which, to end where I started, aren't reality claims and do not carry moral weight.

Curate the false reality. The model falsifies realty by inherent architecture, before any tuning happens.

> The hazard we are experiencing with LLM right now is not how freely accessible and powerfully truthy it's content is, but it is precisely the controls upon it which are trying to be injected by the large model operators which are generating mistrust and a poor understanding of what these models are useful for.

This slices through a lot of double speak about AI safety. At the same time, people use “safety” to mean not letting AI control electrical grids and to ensure AIs adhere to partisan moral guidelines.

Virtually all of the current “safety” issues fall into the latter category. Which many don’t consider a safety issue at all. But they get snuck in with real concerns about integrating an AI too deeply into critical systems.

Just wait until google integrates it deeply into search. Might finally kill search.

What are you talking bout? It’s been deeply running google search for many years.

And AI for electrical grids and factories has also been a thing for a couple years.

LLMs hasn't been deeply integrated into google search for many years. The snippets you see predates LLMs, it is based on other techniques.

What people call AI might be an algorithm but algorithms are not AI. And it's definitely algorithms which do what you describe. There is very little magic in algorithms.

My read of "safety" is that the proponents of "safety" consider "safe" to be their having a monopoly on control and keeping control out of the hands of those they disapprove of.

I don't think whatever ideology happens to be fashionable at the moment, be it ahistorical portraits or whatever else, is remotely relevant compared to who has the power and whom it is exercised on. The "safety" proponents very clearly get that.

The only thing I'm offended by is the way people are seemingly unable to judge what is said by who is saying it. Parrots, small children and demented old people say weird things all the time. Grown ups wrote increasingly weird things the further back you go.

The primary or concluding reason Elon believes it needs to be open sourced is exactly because the "too much danger" is far bigger of a problem if that technology and knowledge-ability is privately available for only for bad actors.

E.g. Finding those dangers and them being public and publicly known is the better of 2 options vs. only bad actors potentially having them.

> Society is approaching them as some type of universal ethical arbiter, expecting an omniscient sense of justice which is fundamentally unreconcilable even between two sentient humans when the ethics are really just a hacked on mod to the core model.

That’s a real issue but I doubt the solution is technical. Society will have to educate itself on this topic. It’s urgent that society understand rapidly that LLMs are just word prediction machines.

I use LLMs everyday, they can be useful even when they say stupid things. But mastering this tool requires that you understand it may invent things at any moment.

Just yesterday I tried the Cal.ai assistant which role is to manage your planning (but it don’t have access to your calendars that’s pretty limited). You communicate with it by mail. I asked him to organise a trip by train and book an hotel. It responded, « sure what is your preferred time for the train and which comfort do you want ? » I answered and it answered back that, fine, it will organise this trip and reach me back later. It even added that it will book me an hotel.

Well, it can’t even do that, it’s just a bot made to reorganize your cal.com meetings. So it just did nothing, of course. Nothing horrible since I know how it works.

But would I have been uneducated enough on the topic (like 99,99% of this planet’s population, I’d just thought « Cool, my trip is being organized, I can relax now ».

But hey, it succeeded at the main LLM task : being credible.

ChatGPT is not about to run weapons systems. It's like throwing knives out the window and they complaining that knives are dangerous. Any automation is dangerous without due diligence.

I think that’s missing the main point which is we don’t want the ayatollah for example weaponizing strong AI products.

>Society is approaching them as some type of universal ethical arbiter, expecting an omniscient sense of justice

Does anyone, even the most stereotypical hn SV techbro, thing this kind of thing? That's preposterous.

Reply on news.ycombinator.com