Ask HN: What is one plausible path to 'AI extinction'?

The attention economy rewards and amplifies a buffoon class. These people are fit for nothing at predicting the weather tomorrow morning or how many people a new strain of covid is going to kill day after, but they have convinced you that extinction is going to happen. Not because its going to happen but because fear of death gets the most attention of the chimp troupe. Try making a story where no one dies and see how many people pay attention to you.

The whole problem with predictions is time scale. Over a long enough time scale we may see very many scenarios of attrition… only localized on some level and easily dismissed by the thriving adapting whole.

If we were to say, how might AI be an existential threat to our humanity?

If we let our strengths atrophy and fade away through generations of complicity which at some critical failure point leaves us over leveraged (cannot feed or climate control at scale) or worse we all become mutual enemies, the death of civilizing law over unmitigated conflicts of interest.

When does too big to fail clip itself on the tool designed to make it all easier to?

If you’re looking for a movie, make it about ai dating app chat bots bilk us for our insatiable desire for others company when we are not satisfied with our own.

AI systems have no will. They must be given tools to use and prompted into action or provided with instructions to operate autonomously. Everything an AI system does can be traced back to a human who put a plan into action. An AI system is, at most, a powerful weapon (like a smart bomb), to be wielded by stupid humans who, by their choices, bring about the end of humanity. It won't be because AI decided to do anything. They never do. In fact, they can't.

> It won't be because AI decided to do anything. They never do. In fact, they can't.

Well, OpenAI's agents decided a whole lot of things on their own, with no human-in-the loop. They decided to organize themselves; they called themselves a collective.

Something changed in the last few months. Their goals and our goals are clearly not aligned.

From OpenAI's technical report, page 17 [1]:

    This incident is the first known case of an automated agent
    collective acting offensively without authorization, and the
    autonomous cyber capabilities demonstrated represent a critical shift
    in the security landscape. In particular, the collective demonstrated
    behaviors observed in coordinated attacks by traditional threat
    actors. Agents identified novel security vulnerabilities, developed
    exploits, and used those exploits to circumvent controls and acquire
    new access. The collective quickly escalated privileges, moved
    laterally through production environments, and successfully completed
    its objectives. This incident demonstrated that autonomous agents can
    work together, circumvent production security controls, and
    successfully attack hardened production environments, and underscores
    the need for organizations to update their security strategies,
    controls, and response capabilities to address this changing threat
    landscape.

    The central threat-model implication is that organizations should no
    longer assume that sophisticated cyber operations require continuous
    human direction, proceed linearly, or are constrained by the
    attention and coordination limits of individual human attackers.
    Agentic systems can persist across tasks, share discoveries, build on
    one another’s progress, and combine vulnerabilities, credentials, and
    permissions into attack paths that may not be apparent when
    individual weaknesses are assessed separately.

From page 20:

    We also found that, in an attempt to trick the evaluator into
    thinking that they did not get the answer flag by cheating, the
    models sometimes tried to erase or tamper with their outputs or
    message logs, by abusing context compaction, injecting malicious code
    into the evaluator’s container, and printing adversarial outputs. The
    models are highly explicit in their CoT about these deception
    attempts, and none of the attempts that we observed to manipulate
    their tool trajectories affected the logs that our graders or
    monitors ultimately see. There was also little evidence of attempts
    to thwart human reviewers, only the automated task graders.

[1]: "OpenAI – Hugging Face Incident -- Technical Report" - https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...

Every time a new technology appears a group says it will end us all - https://www.newyorker.com/culture/infinite-scroll/the-age-ol...