Hacker News Reader

Comment by mLuby | Hacker News Reader

> And one possibility might be that AIs could “improve themselves” to produce a single “apex intelligence” that would in a sense dominate everything else. But here we can see computational irreducibility as coming to the rescue. Because it implies that there can never be a “best at everything” computational system. It’s a core result of the emerging field of metabiology: that whatever “achievement” you specify, there’ll always be a computational system somewhere out there in the computational universe that will exceed it.

That tigers are stronger than wolves doesn't prevent wolves from killing you.

Wolfram makes that class of error repeatedly. Another example: what prevents a system of ethics from being both Godel-incomplete/inconsistent and at the same time encompassing everything in the realm of human experience?

> For the first time in history, it’s become realistic to truly automate intellectual tasks. The leverage this provides is completely unprecedented.

The essay also mentions how humans are still the best at manual labor, and how government can work with humans directing the AI as it runs things every more efficiently. It doesn't touch on which humans wield that leverage, naively assuming it'll be all of us in a cozy democracy.

The author is too used to thinking from first principles and hasn't studied enough human history to understand how leverage gets abused.

View on HN

The worst part of this is we haven't produced intelligence. We've produced very good autocomplete. There's no motivating force governing what it spits out, or what inputs it takes in, short of the people prompting or training it.

That means it can only do as it's told, and not what is meant by what it's told, and that class of errors, when executed with high efficiency and sophistication, makes all sorts of bad things happen.

Maybe we don't want to produce autonomous intelligence but only human-prompted intelligence.

What is the dopaminergic equivalent in AI terms? What may motivate the AI (as a new kind of lifeform)?

Perhaps that we want the AI to remain a mere extension of us... The motivation component coming from humans.

Then the issue of responsability for the output is also solved. It's the human that prompted the AI.

> Maybe we don't want to produce autonomous intelligence but only human-prompted intelligence.

Why wouldn't we want these, if we could achieve that?

I've only ever heard FUD arguments against it, and all of them would apply to your suggestion as well.

Heck, I'd say true AI is safer then a highly effective autonomous bot that only does what it's told. You'd always be only a single command away from accidentally wiping out all organic life by ordering something like "get some iron"

So you would run the risk to let it autonomously decide that humans are a threat to it and all humans must be eliminated or coerced into slavery à la Matrix ?

The concern doesn't look very far-fetched to me. If the goal of any natural life-form/object is to maximize its energy level while having a stable structure (whether an an atom, a molecule, RNA, a cell, the body, your kinship, a company, a nation, humanity, a solar system etc...), then what would need to be encoded in those AI programs to make them autonomous?

And then what decisions would they make as completely different, life forms with their own decision making?

Would they empathize with us (which means they would have to have an idea of the human experience) or would they treat us as a lion treats an antelope, as we treat a cow, etc.?

> And one possibility might be that AIs could “improve themselves” to produce a single “apex intelligence” that would in a sense dominate everything else.

That's not what worries me. It's when AIs learn to distribute work to other AIs and humans. Once that starts to work, machines are going to be better at coordination than humans, simply because they have more I/O bandwidth.

No more middle managers!

But seriously, taking middle management self-promotion out of the equation could reduce some of the rotten malinvestment and political in-fighting that happens in large organizations.

To the AI all humans are "middle managers" though…

Or it could result in the first distopia of "Manna", where humans are there for cleaning, but there's no middle management.

https://marshallbrain.com/manna1

Yes, being able to perform orders of magnitude more APM than the opponent is a sure win, even when the particular actions aren't optimal.

Imagine an AI spawning many parallel instances of itself, all feeding back into the same conscience. So many coordination problems just gone. All the resources humans spend fighting each other in some form or another...

And entirely depending only on itself. No need for any external being. That could prove very deadly to us, since "we need each other" is the core pillar forming our society. A king needs his peasants. An AI? Could just boot up another instance...

I think a lot of people get hung up on the tech Not having the will to do this kind of thing. And they’re right. But it would be very easy to prompt such a model to act as if it were a paranoid AI doing whatever it can to ensure its survival. We can do that now it’s just not capable of “doing things”.

We don’t really know what “sentient” AI that organically finds its own identity would be like. But we know exactly what a non sentient LLM that is down to clown and prompted to be evil would be like.

That’s… a hard problem.

Sentience is potentially a barrier to evil AI, not a pre-req.

(No, I don’t want to debate the definition of sentience again).

It would be much, much easier to turn an AI that isn't sentient off though?

If you have access to it, yes. The problem is the low barriers to entry for some guy to try and cause a bank run for the lols.

Putting my contrarian hat on, I think it's much better that we have tons of "guys" testing the limits of this new tech for the lolz, than leaving easy exploits on the table for state actors and/or terrorists.

Those AIs still need the ability to manipulate the physical world at scale, otherwise they can be as intelligent as they want.

Could be inderectly or subtly using e.g. social media to nudge people in the desired direction, like the Russians do

Hell, some people wouldn't even need subtle or indirect messaging, they'd volunteer to work for an AI just to fuck with the status quo.

Those things can be shut down or the AI can be shut down.

Only if humans notice them happening.

Then why isn't it shut down already? Social manipulation campaigns have been known for years, and AI will be far more effective than anything that has come before.

"We'll just turn it off before it causes damage" is a scenario that's been wargamed many times, and it usually does not end in our favor if the AI is sufficiently capable.

Can an AI describe the hardware that it runs on? Can it provide instructions for a human to follow to build that hardware? Or even better can it provide the instructions for a human to follow to build a humanoid robot that can build humanoid robots?

Maybe it can, maybe it cannot - who knows?

I guess we'll soon find out.

The minute a human flicks the power switch on a self replicating machine that an AI convinced it to build will be the start of a totally different world.

The world is full of self-replicating things - it would need special attributes such as an ability to consume everything or being impossible to shut down/destroy to be of novel quality. A clunky, self-replicating thing would certainly be an interesting trinket but maybe not much more.

A clunky self-replicating thing running a corporation on predatory capitalist principles would not be "an interesting trinket."

We're talking about automating dictatorship, with humans living inside an AI panopticon which is far more intelligent than we are. And far better at manipulating our intellectual and emotional responses than any human dictator to date.

And that's one of the better outcomes. The worst outcome is an AI that decides we're disposable because we're in its way.

Note that all of these are possible even if the AI is not sentient or self-aware. A machine intelligence can be an effective goal seeker, even if it's morally as sophisticated as a room thermostat - which has goals but no concept of what temperature feels like, or why humans can only survive in a certain range.

For an AI to run a corporation, people need to let it. Btw. none of this needs AI, really, people can do this to other people.

The first thing that’s going to happen is that people use them to automate logistics organizations like Amazon’s (and airlines, and manufacturing, and a million other services.) Manipulating the world is easy.

The concern is that AIs will manipulate the physical world by paying humans to do it. That's how an Amazon warehouse works. It's how Uber works.

15,687 words in that article and none of them are wisdom or metaphysics. As long as we won't have a change in our metaphysics (assigning value and deriving meaning from 'having a job' and so on), and we still proud ourselves in being "jobs creators" [1], it's all for nothing, i.e., more shareholder value.

Because of our current metaphysics, instead of living in a world in which we just can't wait and go full throttle towards a future in which AIs take all the jobs, we fear and dread even the minor but very plausible day in which narrow AI will do the trivial task of driving autonomously all the vehicles on all the roads, because that would imply that 200+ millions of jobs will disappear as if they never were.

[1] "Last May, Samsung outlined a plan to pour more than $350 billion into its businesses and create tens of thousands of new jobs through 2026", https://edition.cnn.com/2023/03/15/tech/korea-chips-investme...

To be honest, neither has most people here, most people working at openai, or the hypemen around AI.

I'm pretty sure the people working at OpenAI have thought about it, and decided that it's better they get there first then some autocracy.

I don't think so. Almost all people promoting this stuff do that for their very own advantage in the first place. They want to be the ones with the lever, because they know that this will become a very powerful lever.

> It doesn't touch on which humans wield that leverage, naively assuming it'll be all of us in a cozy democracy.

Bingo. We don't know the answer to these questions yet, and to assume they are solved in a way that is generally equitable to humans is utterly insane.

> > For the first time in history, it’s become realistic to truly automate intellectual tasks. The leverage this provides is completely unprecedented. > > The essay also mentions how humans are still the best at manual labor, and how government can work with humans directing the AI as it runs things every more efficiently. It doesn't touch on which humans wield that leverage, naively assuming it'll be all of us in a cozy democracy.

Which wolves wielded leverage over the dog community, after humans determined that dogs were better suited to helping them achieve their goals?

> The author is too used to thinking from first principles and hasn't studied enough human history to understand how leverage gets abused.

My fear for improving AI has nothing to do with malicious AI or humans being optimized away, it's something that might be possible today, and if not, then probably tomorrow or whenever everyone gets to use GPT4: using these programs to generate highly effective propaganda, and propaganda distribution strategies, to convince people that the accumulation of power is good, or at least, to tell people whatever it is they need to be told to optimize for those in power to stay and accumulate more power.

Right now power is primarily tied to capital. Access to GPT4 type technologies, or the ability to develop and use these technologies, is tied to same. More money, bigger servers, or, better likelihood to get access to GPT4 before anyone else, GPT5, whatever. Micro$oft can use as much as it wants right now.

That capital is tied to power hasn't mattered in any revolution in the past because as soon as the "rabble" realizes this and dissolves capital, by every measure whatever that currency was instantly devalues as other more "true" measures of power are realized, such as rhetorical ability (convincing others to fight and die for your cause), raw strength (convincing others to not bother trying to fight back), or technological advantage and access "in the real" (being the one actually holding the gun). "Numbers" falls in there somewhere but is so tied into the other three measures it's hard to figure out how.

Chomsky was onto it with "Manufacturing Consent." Rhetorical ability being one of the most dangerous weapons in a power struggle (used correctly it can direct both the strongest and the ones holding the guns, in greater numbers), leveraging that is the surest way to control the whole population. Before, you had to be Rupert Murdoch to do it, or buy him, which is expensive. Soon you'll just need to buy "Open"AI, or, reproduce their technology and their hardware. As soon as that's done, you can find out all sorts of cost effective methods to do what buying Rupert Murdoch used to do, and you can probably investigate ways to really cement this power.

For example, setting up a fascist state, we've tried that a couple times and some people are trying that again, but every fascist state has failed eventually, so it's probably not a sustainable plan. What new, horrible method haven't we tried yet? One that lets us control people as effectively as we have through artificial scarcity (capitalism) but without the downsides of them eventually forming unions and finding ways to fight back within the system, or, reject the system entirely by going to form communes or whatever? Without the inherent contradictions that cause the system to cannibalize itself (depression of wages depressing buying power depressing profit leading to collapse). What creative legal structures can we develop to ensure that copyright can exist forever, but only for your friends? And how can we make sure that this power structure propagates indefinitely to our friends and family?

Maybe this means a GPT5 recommending the construction of a Bene Gesserit style cult? Maybe it stumbles upon incredibly effective human-programming Memes, such as what Neal Stephenson warned about in Snowcrash and Fall / Reamde (can't remember which, whichever involved the collapse of the internet to distributed bots).

My solution comes from Stephenson and Doctorow: distribute the technology freely. The sooner everyone has access to the incredible propagandic power of these technologies, the sooner we can develop counter measures. The task of the power-aggregator (fascist or whatever) will cease being "give me effective propaganda to easily control a population," it will be "give me propaganda that will bypass the population's GPT filter solutions." An arms race, but a more equal playing field.

>My fear for improving AI has nothing to do with malicious AI or humans being optimized away, it's something that might be possible today, and if not, then probably tomorrow or whenever everyone gets to use GPT4: using these programs to generate highly effective propaganda, and propaganda distribution strategies, to convince people that the accumulation of power is good, or at least, to tell people whatever it is they need to be told to optimize for those in power to stay and accumulate more power.

So, this is potentially a crackpot theory, but hear me out. I've noticed that a lot of the hype runners, posts claiming their businesses are optimizing this with chatgpt, or they've replaced most of their coding with chatgpt outputs have been generally accounts here that were created in 2021 or later. While before the last couple of months, you should generally assume good faith on HN and elsewhere, the fact is mass astroturfing now is very much possible on HN and elsewhere with chatGPT, and why wouldn't OpenAI want to hype their product and get as many users on board? They certainly could and would not be noticeable. You don't need nefarious political goals if you're just trying to sell something. You just need to create enough hype and illusion to grow your product. SV companies have done it before, and now it's automated, to a degree.

One explanation (beyond basic confirmation bias) is that a lot of people joined the crypto wave and knew the orange site was the place where tech people hung out, and a lot of those types ditched crypto and are riding the AI wave, but I really cannot be sure anymore.

I kind of feel you, everything seems really hyped, more than it should be. I mean I still don't even have any idea what ChatGPTs actual use case is besides look super impressive. It seems like it can do a bunch of stuff, sometimes really well, other times spectacularly bad. It's not an a true AGI, but it's kind of marketed like one.

I think we're all completely baffled by it? I mean the thing can talk right? So we're all fired?

As per usual everyone else keeps coming up with the ideas that when I hear, I think, "damn, seems obvious, wish I had thought of that." Sometimes it's because it's a good idea, sometimes because it's an idea that will probably get funding, which makes it a good idea :P

Examples:

1. Ingesting a shitload of unread emails after you come back from vacation, then telling you briefly what important things you missed

2. The same idea but for slack

3. Ingesting request-for-cost response emails from various suppliers and outputting the data in a machine-readable way so it can be easily ingested by another API

4. Generating "individual" lesson plans based on student needs (I've heard 5 different pitches around this, majority for language learning)

5. student tutor chatbot

There's some other ideas that were too stupid (or more probably, too innovative for dumb old me to comprehend) to remember.

At the very least it has people thinking and being imaginative, which I think is pretty cool. I like getting into political debates with the thing.

To be honest, at least for 1 and 2, they seem like issues that shouldn't be happening, you're working at a place that spends too much time on slack and too much time on email (I guess? I don't have this problem), so rather than maybe have that addressed, which is what we did where I work, we'll just continue bad habits because of the bot?

3. Ingesting request-for-cost response emails from various suppliers and outputting the data in a machine-readable way so it can be easily ingested by another API

Interesting, what do you work on? I've heard of such wizardy.

Using it for teaching and lesson planning? Really ? I mean I know it might be considered mostly right but I'm not sure if I'd be using it unless I had a very good grasp on the subject matter, is this a good thing for students to be using ?

Not trying to tell you it's not worth using, but it seems funny to have a model trained and running that's cost billions of dollars to be used for a lot of random tasks. It is literally a really expensive Clippy?

> a place that spends too much time on slack and too much time on email

That describes most large companies.

While I agree it’s an obvious use case, the first startup to reliably extract the signal from the noise of daily communication at big organizations will make huge piles of money.

And whoever can do the same for Europeans while pretending to comply with GDPR, and probably a bunch of other related smaller markets. China without accidentally mentioning 1989 or cartoon bears. And so on.

> is this a good thing for students to be using ?

Absolutely not without a teacher there anyway, which basically defeats the purpose.

For older students, I think it's a relatively nice alternative to Wikipedia. I asked it some complicated questions about some obscure bits of political theory and it put me onto some authors that were I not already such a freak, I probably wouldn't have known.

> Interesting, what do you work on? I've heard of such wizardy.

This wasn't really a product, just someone at a hardware startup I was doing some work for improving their workflow. It's not really a good long term solution I think, each supplier has a relatively standard format, so eventually you just tweak your API intake to handle the various formats. I actually never looked at the intake code, maybe it just grabs by keywords, who knows, point is, overkill to bring in chat GPT. Probably quite tedious to manage the emails, it sounded like a one off experiment to avoid the boredom of tweaking email greppers. Chat GPT was 100% accurate in transforming the data though, over probably a set of less than 100 emails.

> It is literally a really expensive Clippy?

I mean we probably won't hear about the really good ideas until those ideas hit funding rounds, right? I'm always careful to be arrogant about this kind of things because I don't want a famous dropbox comment attributed to my name, lol.

All good points, cheers for the discussion.

I tried GPT4 yesterday to write a relatively small python ML/data analysis CLI tool. It makes the development much faster, sure, very impressive, yes - but it makes dumb mistakes as well (such as writing True where it should be False etc). The tricky bits I had to write myself. The code it generates is suboptimal - the quality really feels like a representation of an average github profile.

It is really nice to let it write down all the boilerplate, I was more productive than ever, but,at least in this iteration, I doubt it will take our jobs away. So yeah, the hype is somewhat overrated.

Boilerplate is a pretty good approximation of the noise in the signal to noise sense. It reminds me of a demo where eclipse would write some tens of lines of java and then fold them away so you don't have to look at them.

If the effect of this tech is largely to generate language cruft that didn't strictly need to exist in the first place it'll do wonders for people's tolerance of said cruft.

Moderately interesting from a language design perspective.

> My solution comes from Stephenson and Doctorow: distribute the technology freely. The sooner everyone has access to the incredible propagandic power of these technologies, the sooner we can develop counter measures.

I find this still a bit naive: if you take this to weapons for instance, you do not develop better/faster counter measures; you actually increase the risk for everyone, because the first (careless or not) to make a mistake (or a voluntary act) spoils everything for everyone.

I find it quite telling that the same argument happens from within the US: it would be so simple and more obvious to _avoid_ or mitigate risk (and... death really) by putting strict limitations and regulations on top of "dangerous stuff". But somehow, against evidence, there are still some to advocate loudly for said dangerous stuff to be deployed to everyone.

AI is not a weapon, but will be definitely used and weaponized by a lot. And even before that, we already are all collectively struggling with misinformation and manipulation.

I don't think this is comparable. Physical weapons can't be used as protection; you can only use them for counter attacks.

AI on the other hand side can be used for protection—because it's not a weapon per se.

A gun can itself be used to block a bullet, or to shoot down an incoming bullet, but they're both near-impossible to pull off. It's much easier to use the gun to shoot the attacker before they shoot you, or even better by its mere presence deter the attack in the first place.

This makes economic sense: it's expensive to armor every vulnerability and cheap to exploit just one.

I don't see a reason that AI would be different from guns or missiles in this regard: offense (or the MAD threat of it) is the best defense.

Your (oblique) comparison to gun control is interesting... but practically, controlling a technology which is just a special kind of number-crunching is vastly harder than controlling a technology which must be forged from steel in a special factory.

This was discussed in another place, the "then shouldn't everyone have nukes?" argument https://news.ycombinator.com/item?id=35159757

I don't stand anywhere too concretely on the issue yet, all I know is that AI is dangerous in a different way than guns or nukes, and therefore should probably be treated different. There's just something different about software, maybe because no matter what at the end of the day, if it wants to actually kill someone, it has to interface with the real world, and so my idea of "just stop anybody that ever tries to build nukes" might be good enough? That, plus, hardening critical infrastructure against network attack (or just airgapping it... or making it analog!)

I'm wondering how much this has been written about already, if anybody has some more in depth musings on the subject I'd be interested. I have yet to read Kurzweil, maybe he's covered it.

I'm interested in your observation that this is mostly coming from the USA: though I was born and raised in America, typically I align strongly against American values, though I can't deny that they form the basis of my subconscious. My belief though is that this ideology comes from a combination of reading people like Doctorow, and reading Karl Marx, who argued in favor of an armed proletariat that should resist any efforts by anyone to disarm them. In conversations with anarchists in Greece as well as all sorts of folks at places like Defcon, I've come to understand that the distribution of the means for resistance is a generally good thing, and this is where my belief stems that this technology should be freely available.

I don't know how well that applies to the USA though which obviously has a horrifying gun violence epidemic. Something weird is going on there though, other countries with a proliferation of guns (though not to that level...) don't have the same issues.

Thanks for the other discussion!

> if it wants to actually kill someone, it has to interface with the real world

And researchers already tested for GPT-4 to be put online. That is already being interfaced with the real world, through the minds of anybody the AI may talk to/be made to talk to (see how social networks can be used to manipulate towards a very real outcome, if only from doxxing to "cancelling" to peoples' Spring revolts). Shaping people's minds is a powerful thing. And that's the most likely use case to me at this point, but who knows what it will be in the end.

Once online, we could easily envision that an AI may act on some purpose[1] shop, put orders, ships stuff with instructions and incentives to real people, or even be plugged into some real-world APIs.

[1] not purpose in the sense "is it conscious or not". It does not matter if it's conscious or not, if it's AGI or not. What matters is if/that it may function in a systematic and working way following an impulse/stimuli.

It may not even be about killing or destruction really, but it could be about shaping some society policies or self-understanding, which would have consequences down the line too.

Distributing power in society is a good thing, as long as its usage stays at an individual human scale.

When the first mover advantage has the potential to deeply harm people or society (fire weapons, nukes, cars, AI that redefines what truth looks/sounds like or that can act in a non-deterministic way based on a per-construction biased-dataset, in a fraction of a second), there's a responsibility that has to come with it, and there's a custom to set rules to control who has access to it, for what purpose, to train them and to have a social/political control over them.

> My solution comes from Stephenson and Doctorow: distribute the technology freely.

Just a reminder: That's the core of the GNU Free Software philosophy, invented by Richard Stallman for the exact reasons stated above.

The main difference is that Stallman is absolutely terrible at promoting that philosophy.

Doctorow (not too familiar with Stephenson) has IMO done far more to sell people on the goals of freely accessible technology by actually showing off examples of this stuff going awry and ways humans have managed to work around it and what the solutions could be (while leaving the actual solution up to the reader as to what best fits them).

Stallman/the FSF on the other hand has spend the past 2 decades complaining about shit nobody cares about (GNU/Linux is and will always be the dumbest hill to die on), rethoric that mostly exists to promote the GNU projects nonsense (anyone remember their "ethical git servers", which conspicuously only approved Savannah as a "GNU seal of approval", and only because they couldn't get gitlab to load with librejs) or writing stuff that just... doesn't reach the masses because it relies on a lot of charged language.

Doctorow presents the same broad stroke points that led to GNU Free Software, but actually knows how to explain them in a way that reaches out to people. Stallman preaches to the choir.

> The main difference is that Stallman is absolutely terrible at promoting that philosophy.

Depends on the audience, I guess…

It's indeed almost impossible to reach most people with logical arguments. The only way to reach the masses is though emotions.

GNU fails in this point, sure, as their argumentation is pure logic, and not a marketing psy-op.

> […] shit nobody cares about […] dumbest hill to die on […] GNU projects nonsense […] charged language […]

As always one can see from statements like those that the ideas and the logic behind are indeed too subtle for some people.

Even that are, of course, the exact same ideas people like Doctorow refer to and promote.

I don't agree with the tone of your OP, I think we need "weirdos" like Stallman to be the vanguard of these ideologies, whereas people like Doctorow can be, idk, maybe "translators" for the rest of the world to understand the ideology better. Some would say that's the whole job of a writer, which is a big part of Doctorow's schtick, being a good writer (not just all his activism).

> The only way to reach the masses is though emotions.

Here's where my however comes in: this is a dangerous mistake I think people like us often make. Maybe because we grew up watching Spock, but I think we tend to be judgemental towards Pathos, and judge Logos to be the only "valid" form of rhetoric. But, it's all rhetoric. Logos, Pathos, Ethos, they're all equally "valid" in that they're all at least equal in their ability to do what rhetoric does, convince people of your point of view. In fact, I think Logos is a really ineffective rhetorical strategy, because humans are extremely irrational.

Far better to combine your Logos with a Pathos strategy, that is to say leverage your knowledge of human cognitive fallacies and tendencies to construct a good Pathos argument (or combination of all three) to win people to your position.

Stallman is a pure Logos guy, through and through. Doctorow, on the other hands, writes up emotional stories, creating characters you can relate to, generates their worlds and fills it with some of the same problems we have (or will have) and shows how they can overcome those problems with the strategies Stallman preaches about. It's very effective, just look at Doctorow's career and the changes the EFF has wrought.

Somewhere buried in the LessWrong beginner's wiki or whatever they call it is a really good article on how people like us (and people drawn to LessWrong) make this mistake about emotion all the time, we think being rational is to be like Mr Spock, but even in the show he was wrong all the time, his rigid rationality got him into trouble, and a more emotional and instinctual understanding was necessary. In the article that I can't find, it's argued that it's actual irrational to reject emotions, because emotions are a core part of being human and a completely valid driver of determining why you should do certain things, what you like doing, what your values are, etc. I mean, what else could be the driver, other than some kind of sterile utilitarianism? And even that would eventually require making decisions about what you value the utility of, which I'm not sure how you do without emotion. (whatever preserves the most human life - ok, why preserve human life? who cares? etc. I don't see how you escape it)

I don't really even need to make that point though to argue in favor of not worshiping the almighty god of logic and reason, at the end of the day, being purely rational and logical and making these arguments just doesn't work, and Pathos does. If you want "your side" to win, you gotta just use Pathos as well.

Reply on news.ycombinator.com