Pack Hunters of the Silicon Savannas | Hacker News Reader

Pack Hunters of the Silicon Savannas

hintjens.com

89 pointsedward34 comments

tldr: malware evolves into superintelligence. I'm afraid evolution doesn't work like that. Here's a good book (some math):

https://www.amazon.com/Evolutionary-Dynamics-Exploring-Equat...

The long and the short of it is that software can't evolve because too much of its genome space is lethal, and the non-lethal parts are non-contiguous, so the quasispecies decoheres -- if it survives at all -- instead of climbing to the top of the fitness landscape.

I'd also suggest a book by Axelrod, The Evolution of Cooperation. [1] Not software explicitly, but patterns of behavior, which is pretty close.

[1] https://en.wikipedia.org/wiki/The_Evolution_of_Cooperation

The "Tit for Tat" in the Axelrod experiment winning out is probably the clearest example of a global-winning strategy that's still susceptible to a single-hit psychopath (worse, multiple hits by a sociopath).

The core idea is that lots of nice strategies working with each other can beat the ingenious evil ones, is a really positive spin on the usual Prisoner's Dilemma's "defect first" model.

This is true for classical machine instructions, but what if future software looks more like a huge neural network with a small driver attached?

Modern artificial neural networks aren't designed like that; they are pattern recognizer/generator systems. Further, it is understood that Turing machines and neural networks are computationally equivalent. So it doesn't advance the question to reframe it into a different computational substrate.

I read this article as effectively a singularitian belief statement. Like the Singularity claims, it doesn't make sense with any existing technology stack from a hardware-code-and-bits perspective: no way to get there from here. It's a belief system or a visionary statement, not an engineering reasoning chain.

If I'm understanding you correctly, you're assuming that "software evolution" refers to random mutations in the software's core source code?

If so, I might be inclined to agree with you on that narrow point, but software/AI programs can still evolve in other major ways. Assuming an AI that operates on the basis of Neural Networks for example, it could evolve its number of layers, the size of each layer, its sensitivity to new information, and other such parameters.

It's not clear if the concept of predator-prey makes any sense in the context of software programs, but invader-defender and virus-host relationships could certainly exist in a advanced software ecosystem, similar to in nature.

The premise of the essay seems a little too far-fetched for me to seriously worry about it, but it does seem like a great premise for a sci-fi novel.

If you insert a random nucleotide into a DNA sequence, the protein you get is usually very similar to the old one. But if you insert a random branch instruction into the object code of a working program, the results are somewhat different.

The problem with software evolution is that the adjacent points in program representation space -- whether it's source space or binary , neural network parameters or whatever -- are not viable individuals that can participate in an evolving population. They code for junk -- stillborn offspring. This isn't terribly surprising. The apparatus that powers living systems and makes DNA sequence space so nice took almost a billion years to evolve. Building a such a system in silico, one that is both that flexible and that forgiving, is not likely to happen anytime soon.

But you're right. I'd totally read the sci-fi novel.

> But if you insert a random branch instruction into the object code of a working program, the results are somewhat different.

Maybe, maybe not. I've observed lots of dead code in my time developing, where inserting random instructions would have no impact whatsoever. Inserting random instructions into a give program may have negative side effects, it may not. But (without having read your reference, though I might, or the linked article here), most of the source and object code that exists is highly specified and functionally dense. These are the ways we code when we want something very optimized and not broadly adaptive. Essentially you could say that almost all of the code we are writing is rain man style code. Good at some few sets of things in specific conditions. I think it is possible that there will exist (although perhaps not written in the same way) code where less of the genome is lethal, and there is more "wiggle room" for code to expand and change as a system. The stillborn offspring we might encounter if todays systems were "genetically evolved" are the equivalent of disseminated haploid genetic material. I think the conditions for conception (if you'll ride with me on my beaten metaphor) are not so far off as you may think, even if they are very different in terms of compilation/process execution/parallelism. While much of the code we write still mirrors the serial logic we often apply in the sciences, it is neither a foregone conclusion that software will continue exclusively in that way, nor a lack of capabilities of modern infrastructure to achieve parallelism comparable to some biological systems.

The novel "StarFish" by Peter Watts (and it's sequel) plays with this notion a bit actually.

> I've been running somewhat sadistic live experiments over the last years. The results show that loosely-organized groups of "average" people will systematically beat tightly-coupled teams of "brilliant" people, when solving the same problems, on the same playing field. We have done twin studies: take the same software DNA, fork it, and raise in two opposed environments. The outcome is dramatically clear. The crowd beats the genius. Not just once either. It happens over and over.

I'd like to know more about this. Has he written about these experiments elsewhere?

The specific case was the fork of ZeroMQ in 2011 to create Crossroads.io, in which the two dominant contributors took the project and created a head-on competitor. They used their own preferred process. The ZeroMQ project switched to a community-driven process (C4). Crossroads died after a short while. It then spawned Nanomsg, which tried again to beat the ZeroMQ community in this game. Again it failed. That spawned several dark forks, none of which has had any traction. I wrote about this in http://hintjens.com/blog:112.

I didn't provoke the fork, really, yet when it happened, it seemed like a perfect experiment. Same DNA, two different environments.

I think the sample size may be too small to draw any general conclusions.

This is absolutely true, and why it can only aspire to science at this stage. Many factors were involved, such as network effects.

I think that's a very plausible conclusion. I've been working up something along the same lines (further justifying the underlying concept). The mechanism you're using is essentially the genetic algorithm, and that depends on being fed an extensive diet of 'not brilliant' people to throw semi-complete and individually failing ideas into the mix until you get a lucky combination.

It is indeed science, just maybe not the science you expected :)

This is obviously entirely false if one takes "genius" to mean someone who does things a non-genius would not be able to do.

There have been attempts in recent years to roll back and debase the definition of genius to be more compatible with egalitarianism (especially in parts of Europe) and the tenet of homogeneity but this is not what genius originally stood for.

Fabian Tasano writes:

"Regarding the version of “genius” that is currently in retreat but still occasionally used: many people seem to have a simplistic idea of what it takes to be one. According to one popular model, all that is required is an increase in the magnitude of certain qualities which everyone already possesses in some measure. Make the particular qualities pronounced enough, and you get to genius.

But a better way to understand the concept — assuming we’re applying the word to (say) Gauss or Picasso, rather than John Cleese or Wayne Rooney — may be that a genius has a particular capacity, which on a certain level can seem obvious or unremarkable, but which no one else has.

A genius, on this understanding, is a person uniquely capable of making a leap ‘off the path’. With hindsight the leap may seem simple or obvious, but at the time no one else was, apparently, capable of making it.

A potential leap of this kind is made possible by preceding leaps. Nevertheless its actual occurrence may go on not happening for decades. During that time there may be clear pointers towards it. Yet it is not until a genius comes along that the leap actually happens."

I could not agree more with this. Alan Kay also outlines an extremely similar point of view: https://www.youtube.com/watch?v=FvmTSpJU-Xc (Normal considered harmful).

Programming is an artistic/creative process. The crowd will never beat genius in this game since the "vision" of the crowd will be diluted by each participating individual. The obvious exception here lies in crowds that are ruled by geniuses with an iron fist (thus not really committees), something which can only strengthen my point. Examples that support this from the past 30 years in technology are countless. We even have sayings about software designed-by-committee.

The point Hintjens is trying to make with ZeroMQ is flawed because he equates the evolution of an idea that has already manifested to the inception of the idea itself.

You're simply shifting the goal posts to try to keep others from scoring. There is no "idea". There are hundreds of thousands of small incremental steps.

Apparently, there was a story about Murray Gell-Mann, where his graduate students realized that he didn't think of things they couldn't, he only got there faster. So while exploring a new line of research, they brainstormed for an hour without Gell-Mann, then had their new insights thought out ahead of time. So they then reenacted the brainstorming session with him, such that they could blurt out all of the new insights just before he could speak. (Usually, it worked the other way around.)

He was left wondering if he was losing his touch.

I had a grad school professor who was like this. He was constantly speaking the thought, just as you were completing it in your head.

"Are things better now? For most of us, life is easy. We are fed and pampered, kept healthy and alive. We left our planet with our new owners, and never looked back. Perhaps wild humans still roam there. Disease, conflict, poverty, hunger, waste: these are historical concepts, unknown to us moderns.

Does the immortality compensate for utter loss of self-determination? Did the Borg(s) save us from ourselves, or did they break the branch of human history? When they bred the predator out of us, did they raise us up, or condemn us to evolutionary death? It is unknowable."

In the Matrix movie there was an interesting moment when Cypher grew tired of all real world struggle and wanted to return into his cocoon, take the blue pill back and to enjoy relative comfort in the illusory world of the Matrix. Self-determination, obviously, didn't work for him.

If you like this line of thinking, you will appreciate the recent book "The Seventh Sense" which looks at the upcoming world from the perspective of network science. I'm about halfway through and highly recommend it.

As much as I enjoy fictional, armchair partial-cosmology, this evolution-based explanation of every aspect of humans - intelligence, social behaviour etc, is just so unimaginative. Couldn't evolution itself evolve to a point where it diverges from the conventional structures held to compose evolution?

Besides, if self evolved as a response to cheating, we must explain cheating outside of the concept of self. Why would you cheat if you are not separate as an individual?

"Couldn't evolution itself evolve to a point where it diverges from the conventional structures held to compose evolution?"

In theory, that's what intelligence/civilization is doing isn't it? We're replacing one optimization process (natural selection/mutation/etc) with a new one (forethought/planning/etc). When we talk about genetic engineering, or AI, or anything like that, we're talking about replacing evolution with intelligent design. We're evolution++.

Well, that ending was unexpected, though certainly foreshadowed appropriately.

removed, apologies

thats some serious mud-slinging. would you care to at least explain why you think that?

I regret posting this and will delete it. it is not meant to be mud slinging. I care for him. I dont know him personally. Though i have interacted with him on lists and he is charming and knowledgeable and generous with his time. I just think he went a little bonkers at some point. Read the book and come to your own conclusion.

Hit a little close to home, did it? :)

i think it does for everyone. life is messy and there are few innocents. we should strive to do the right thing, but branding everyone who disagrees or has done injury as a "psychopath" is not on that path. there were a lot of cool and honest insights which people dont usually speak about which i enjoyed, thank you. and there are many people who are dangerous who should be treated carefully. but i came away questioning the state of the author.

Speaking as the author, there were certainly moments when I questioned my own state. That is part of the path, it seems. And indeed, most of us do harm at some point, yet are absolutely not psychopaths.

I thought one of the defining characteristics of a psychopath is that they don't question their own state. When a normal person does harm, they're like "Oh, shit, am I a psychopath?" When a psychopath does harm, they're like "They were a weak and useless person anyways. This is the way of things."

Yes, this seems to be the case. The whole concept of social judgment is alien to a mind that sees society as a collection of meals.

All of society is alien. Artists, who are basically half-aware,half conscious and build stuff with there subconscious. You never get a answer why that pencil stroke was done. Yet it is done.

The behavior pattern to pressure sub-groups by sexuality into pseudo-judicative contract groups, to allow society to form against the howling wolves out there.

The bred in drug addictions, that work as motivators and the deformations of different lives that defines the directions all those glider guns take.

The machines that enhance us, allowing for society to watch itself and control itself. Its a wild world, a wonderful world. Building social machines that are tailored to what we are, not what we want to be, was one of the archievements. For all its understanding- i can not see enough psychopathy in the authors approach to using what is. The Founding fathers of modern democracy at least came farther. They made battling grounds for them called parliaments and markets, and courts to avoid escaping wolves. One should strive for Leviathancybernetics, to enhance the beast that is, not to fight it, as long one has not the tools to change the very nature.

I knew a medical professional who had to deal with a diagnosed, high-functioning (successful entrepreneur in a "respected" line of business) psychopath in a situation having nothing to do with psychopathy (routine physical ailment). Professionals at the institution were afraid of interacting with the psychopath, out of fear of being sued -- hence my obfuscation of the story. The psychopath very well knew that she/he was a psychopath, but saw nothing wrong with that. There were multiple (more than two) divorces, huge history of litigation (including "ambulance chasing", etc...)

There's no question that non-psychopaths are capable of terrible and even predatory behavior (and certainly multiple divorces and frivolous litigation), the difference is shame, and personally recognizing that there's something wrong with the behavior. This is different from being aware that society sees their behavior as wrong or hurtful and then acting in a way as not minimize harms inflicted by society's punishments (e.g., something akin to what was espoused by the Thrasymachus character in Plato's Republic or -- even more ominously and dangerously -- doctrine of legal positivism).

The latter leads to a very sad point that is relevant to those designing anti-abuse systems of various kinds: anti-abuse systems can also be misused as means of abuse by the sufficiently unscrupulous, Machiavellian, and especially psychopathic (and so can the reaction to this abuse...)

I definitely got a Michael O. Church vibe from this article.

Reply on news.ycombinator.com