The Well-Poisoning Machine | Hacker News Reader

The Well-Poisoning Machine

hachyderm.io

71 points_Microft33 comments

Good rhetorical imagery, but ultimately an emotional argument with a subtext loaded with assumptions:

1) that there is an unpoisoned well to begin with

2) that the chemicals are poison or can only be used for poison

3) that increased access to chemicals will result in more poisoned wells

4) that the people who invented the chemicals are benevolent stewards and will not just use them to further their own interests

5) that it is better for aforementioned benevolent stewards to hoard the chemicals than permit access to those who would concoct poisons, antidotes, and preventative measures, not to mention industry innovations that aren’t related to poison at all

6) that the interests of the people on the ethics committees are aligned with the interests of humanity

7) that ethics committees employed by a chemical manufacturing firms is the only thing standing between ethical and responsible management of chemicals and complete chaos and environmental destruction

8) that the fired ethicists were good at their jobs in the first place

This argument does not feel emotional to me, it feels like an intuition pump (https://en.wikipedia.org/wiki/Intuition_pump). Moral arguments almost always work by analogy, and this is one of them.

1) This assumption does not affect the example. If the well is already somewhat poisoned, making it more poisoned is still Bad.

2) The chemicals are definitely poisonous, but also have non-poisonous uses. 'The does makes the poison,' as Paracelsus put it. You still don't want them in your well water, typically.

3) Increased presence of poisonous chemicals in the well is literally what is presupposed by the example -- it's not an assumption, it's a premise.

4) This argument is anti-chemical-in-wellwater, it in no way touches on 'chemical stewards'. It's weird that you introduce this here -- effectively, you're modifying the analogy, and then pointing to the modified analogy and claiming it doesn't work.

5) See 4

6) See 4

7) This is a pretty reasonable assumption, given that there is no 'Ministry of AI'.

8) The fired ethicists may or may not have been good at their jobs, but no ethicists is probably worse than fallible ethicists.

You want to deflate the wallop this analogy packs, but I don't think you succeed.

EDIT @ 19:10PDT: added ', it feels like an intuition pump. ...'

For 1, no, having more poison in a well and therefore making it more murky, green, and scary looking is not worse than having just enough poison in there to kill you.

For 2, I disagree - you may want chemicals in your well if they’re designed to counteract the poison that was already in there. See 1.

In 4, 5, and 6 it seems you’ve lost sight of the fact that the analogy is being used as an argument against something that’s occurring in the real world. Whether my counter argument perfectly fits the original analogy is irrelevant because I’ve merely co-opted it as a rhetorical device. Are you suggesting it doesn’t fit with reality?

For 7 and 8 I’ll just emphasize that the ethicists were employed by the chemical manufacturing companies. A conflict of interest is not a great foundation to build on.

> no ethicists is probably worse than fallible ethicists

How fallible are we talking here? “Probably” is carrying an awful lot of weight in that argument.

> You want to deflate the wallop this analogy packs, but I don't think you succeed.

That’s just like your opinion, man.

N.B. textninja quietly admits my most important point, #3.

> That’s just like your opinion, man.

Yes, a state of affairs I indicated by using the qualifier 'think.' :) Writing is a competition for the opinions of your readership -- what did you take yourself to be doing?

> In 4, 5, and 6 it seems you’ve lost sight of the fact that the analogy is being used as an argument against something that’s occurring in the real world. Whether my counter argument perfectly fits the original analogy is irrelevant because I’ve merely co-opted it as a rhetorical device. Are you suggesting it doesn’t fit with reality?

No analogy can be extended indefinitely, or you wouldn't be working with an analogy, you'd be observing an identity. It doesn't work, rhetorically, to take someone's analogy and point out that it is not `===` with the analogand. That's basically just an assertion that analogy is impossible.

Given the analogical elements introduced by the original toot, it works.

If I liken thee to a summer's day, it is no fault of mine that you have no solstice.

> For 7 and 8 I’ll just emphasize that the ethicists were employed by the chemical manufacturing companies. A conflict of interest is not a great foundation to build on.

No disagreement here. There should be an independent monitoring body, something like the SEC. The SAIC? (with apologies to the School of the Art Institute of Chicago)

> No disagreement here. There should be an independent monitoring body, something like the SEC.

We may disagree even more than you think ;)

I was pointing out the self-own: that your stated position motivates an outcome orthogonal to your values.

Which of course implies that I successfully inferred your accelerationist libertarianism from context. :)

> Which of course implies that I successfully inferred your accelerationist libertarianism from context. :)

I mean, I did use 3 out of 8 points to rag on ethicists.

Wait till you find out Nozick was an ethicist, that libertarianism is a position within political philosophy (which in turn is a generalization of ethics.)

I’m a programmer, not a philosopher, so I never actually heard of him, but from his Wikipedia entry I learned that “he rejected the notion of inalienable rights advanced by Locke and most contemporary capitalist-oriented libertarian academics,” and let’s just say it didn’t say anything about him getting fired from his job.

> N.B. textninja quietly admits my most important point, #3

Your most important point is the distinction between an assumption and a presupposition? I honestly didn’t see the need to challenge that.

*premise. Yes, this is basic Logic 101 stuff. Not even higher-order logic.

If you mistake an explicit premise for an unstated assumption, and shout 'gotcha!', you're in the weeds.

We’re on two very different wavelengths my friend.

I think what’s happening is that I’m arguing between the lines and you’re arguing on them. It’s like matter and antimatter, but no matter.

Rather, you're not arguing using coherent logic, you're making a series of logic-ish maneuvers that yield persuasive emotional appeals -- which, yes, is perfectly valid rhetoric-work, provided you haven't taken a knife to a gunfight.

You say you're 'arguing between the lines.' Do you code between them, as well?

> You say you're 'arguing between the lines.' Do you code between them, as well?

Wouldn’t it be impressive if I did?

no; that's how we all begin.

> Yes, this is basic Logic 101 stuff. Not even higher-order logic.

Best practice is to forget everything you ever learned in a 101 class. And don’t bring logic to a word fight.

The only good ethicist is a fired ethicist.

Relax, I’m being flippant, but think about it.

Pitch-perfect.

To the mis-en-scene, I could only add: the erotism that is behind technological accelerationism.

What I mean is that there is a sort of visceral, cthonic attraction to the idea of creating 'new life', even as the ecosystems that support actual living systems grow frail. Just as Oppenheimer made the bomb in part because he couldn't resist the attraction of solving the intellectual problems inherent in its design, we're poisoning the well not just to do it first, but because we are hot for well-poisoning.

It's such an interesting intellectual problem! Poisons are beautiful.

After typing this, I realized that there's a great movie that captures this exactly: _Dr. Strangelove: Or I how I Learned to Stop Worrying and Love the Bomb_.

The eponymous character's boner-like n*zi salute, irrepressible when discussing nukes, reminds me of the irrepressible boner-like response that the most brilliant minds I know have to GPT-4.

Me included.

The other cultural touchstone the current vibe reminds me of is Saruman telling Gandalf about the inevitability of Sauron's reign. That same glitter in the eye: "We have work to do."

Mr. President, we cannot allow a mineshaft gap!

Could someone who is actually good at such things please deepfake me Sam Altman's face atop the dude who bullrides the falling nuke in the film's final scene?

Ask Sam's AI to do it for you.

Now think of the current generation of AI programmers who were born after the Cold War ended, who don't understand Dr Strangelove.

This is a problem that I hope attracts much more scrutiny in the near future, and beyond the domain of what's called "AI" right now. It doesn't feel talked about much at all. It feels like we can analyze the inner workings of new technology endlessly, because shiny new tech is cool and racks our brains, but hardly ever stop to consider why we keep pressing on to make even more advances. "Because we can" isn't a satisfactory answer, especially not when the stakes are so high. Why do we keep saying that? Can we ever be totally satisfied with what we already have available to use? Does this imply there will never be a stopping point? Would we go beyond inventing AGI if such a thing were possible?

I think the idea of resistance to change to preserve our chances of survival is worthy of study. I hope there will be serious anthropological studies investigating the nature of that kind of resistance and the opposite, driving insatiability of unbounded, overextended curiosity that marches forward no matter the cost - even if the chances of it affecting the final outcome are slim. It doesn't seem like there's much of an alternative besides repeating "pandora's box, cat out of the bag, genie out of the bottle" as a post-hoc justification for every set of unintended consequences for the rest of time.

The fact that we have imaginations and can foresee the potential consequences of actions has helped our species for so long up to this point. But if safety regulations needing to be written in blood to be effective is really the way our species is most likely to operate, then that has terrifying implications. I think 10 years from now a television crew will sit down the original researchers and engineers behind this tech branded AI and ask: Why didn't you stop?

Ok. I'm dense. I can't understand what this is about. Is this a threat?

I read it as a way to express the frustration about the speed with which AI tools are unleashed without proper consideration for the consequences 'just because we can'. We can't allow a well poisoning machine gap to come into existence.

Seems like a flagrant strawman to characterize AI as a “well-poisoning machine.”

Let's wait and see. I can see plenty of ways in which that could come true.

It's not a strawman, it's an analogy.

In order to be a man of straw, it would have to claim that AI "just is" a well-poisoning machine (in the literal sense of a machine that inserts poison into a hole in the ground containing H2O).

Now, that alone does not mean that it goes through; the analogy works as an intuition pump, and those can be flakey (see https://en.wikipedia.org/wiki/Intuition_pump)

But nor does it mean that a fallacy has been committed

Not AI as a whole. Particular activities to do with AI which would generally be deemed not only dangerous, but also not having any clear non-malicious purpose.

So if there was a strawman argument here at all, it's literally the one you just made!

This is about Sam Altman types rushing to win the ML race. We can reasonably expect civ ('the village') to be disrupted ('poisoned'), but instead of that counting as a reason to not do it, it counts --- somehow --- as a reason to do it harder, so we do it first.

OpenAI is, seemingly, trying to have it both ways: on one hand, they use warnings about the dangers of AI to justify their existence and practices such as keeping models closed, etc. And on the other hand, they're rushing towards commercialization in order to beat out their rivals.

It's a bit of a BS act. Either it's bad and we should slow development way down until our frameworks for dealing with it catch up, or it's not going to turn us all into paperclips.

(I'm one of those boring folks who think there's a lot of risk but not in the paperclip sense but in the "when we use tools we understand poorly, we are likely to get outcomes we may not desire in terms of fairness, trustworthiness, etc.")

The machine, in any case, is well poised to change the world.

Reply on news.ycombinator.com