> stars suitable for settlement should provide “environments nearly identical to that of the home planet.”
Seems pretty clear that we're on a path to machine intelligence, and that interstellar voyages will be undertaken by those intelligences, not meatbags. That humans like a class G sun is irrelevant to the prospering of those intelligences.
Nothing around me keeps working without human interaction for even very small timescales.
Ah, but the trick is, as soon as you can get a machine intelligence smart enough to work on programming tasks, you can get it to help you design a better one. And so on, etc.
The first AGI is (some) amount of time away. A more sophisticated AGI is coming not long after that, and exponentially faster and so on.
There is no law of the universe that says it has to be possible to generate human-level sapience with less than one kilogram of processing mass. In all likelihood we are never going to create a server with a whole city of intelligences in it, ever.
Natural selection is
- random
- blind - gradient descent is prone to getting stuck in local minima, has no foresight, no ability to go back to the drawing board, no "understanding" of what it's doing.
- not even optimizing for intelligence, except instrumentally - to the extent that increasing intelligence interferes with survival, it has to be sacrificed
- subject to hard constraints like the limits on size imposed by childbirth
A computer with five times the mass, five times the volume, and five times the power requirements of the human brain, that can run an uploaded copy of a human brain at merely half the speed of a human brain, would still render Pluto and Mercury trivial to colonise with machine intelligences.
And if it cost a $100k (inflation adjusted) to build, and lasted 80 years before you had to melt down the hardware and reforge from scratch, it would still be cheaper than Musk's target for a human to Mars.
Mere COVID disrupted that here; imagine what a few hundred million miles of space would do.
Much of the plastic supply chain starts as oil or gas. Wood is a major building component as well.
https://www.researchgate.net/publication/322275580_Electroly...
> Much of the plastic supply chain starts as oil or gas. Wood is a major building component as well.
If we really needed those things (why would we even want to use wood as a structural element on a space habitat?), we can build bioreactors from metal, glass, and water.
Oil and gas do occur non-biologically e.g. Titan, but they can also be made from algae grown in a transparent tube exposed to sunlight and provided with the necessary minerals and CO2, which is trivial to make.
IIRC the two limited resources if you needed biology and couldn't do it all with a clanking replicator are nitrogen and phosphate, everything else is easy to find basically everywhere.
The synapse is an incredibly complicated structure and there is a lot of 'computation' occurring within the butons. We're not very sure of all the processes that occur at this time. It's not just a 0/1 kinda thing. Also, our current computers aren't even running the hardware that a synapse is. The most analogous electrical structure to a synapse is a memristor, something we can't make at scale right now. The synapse is also not the only structure that causes computation to occur, many things effect the firing of a neuron and that modulation.
Great lower bound calculation though. I'd say that's the right ballpark number.
Another is that Neurons are incredibly slow. Lets say you can make a machine only half as intelligent as a Human but that can think 1 Million times faster, this is even a lowball since a neuron can fire at like 1Hz and modern CPUs operate in the GHz range.
>In all likelihood we are never going to create a server with a whole city of intelligences in it, ever.
Think this is a really bad take. If one assumes even a .1% improvement in "Machine Intelligence" per unit time then it is literally only a matter of time until you reach Human level and then pass it.
This comparison doesn't make much sense - computers have a very small number of cores, let's say 10, and brains have 86 billion neurons. 86 billion things operating at 1Hz is also in the GHz range. This is leaving aside the issue that a CPU cycle and a neuron firing are doing completely different sorts of work - comparing them is kind of nonsensical in the first place.
>This is leaving aside the issue that a CPU cycle and a neuron firing are doing completely different sorts of work - comparing them is kind of nonsensical in the first place.
Maybe, but the op comment was explicitly using raw transistor count to compare to Neuron count as a proxy for sapience.
That's like saying "even if there's only a 0.1% improvement in materials strength per unit time eventually we'll have structures strong enough to bounce off an incoming rogue planet." There is an upper bound to computational performance per unit mass and unit energy. We're nowhere near it but it is there, lurking.
Note that I am extremely skeptical of your 1T parameters claim - there is much much more to AGI than natural-looking natural language processing + image recognition.
you jump in the air at high noon. the sun is (some) distance away, and for some time, you're getting closer to it.
if we make an ai that's able to make a better ai than we are, then it does stand to reason that it'll kick off exponential growth for some time, but it'll reach limits just like anything. those limits may fall short of magicing/smarting away the obstacles to colonizing space.
and we'll reach limits too. it might be that the amount of time between now and when we make a sufficient ai is more time than we have.
A human jumps, fails, learns the underlying physics of what’s holding them back, and consequently build rockets to sit in. There are no hard limits yet, as far as we know, and we’re just bags of mostly water.
Whatever the hard limits on AI, they will be so far beyond us that we won’t be able to visualize or guess at them.
Computers might be a better analogy. We started designing computers without the aid of other computers. We made very rapid progress on architecture and semiconductors. We’re now completely dependent on computers to help us design future ones, and despite all our efforts and hundreds of billions of dollars in investments, our rate of progress has slowed to a crawl and we seem to rapidly be approaching the end game. Dennard scaling has been done for 15 years. We can make wires that are smaller but just proportionally worse. We can use more silicon area, but then latency and power consumption get proportionally worse. A super clever hyper optimizing AI might push a couple things forward a little bit (or more likely wind up at the exact same destination slightly earlier than we other wise might have), but there’s no compelling reason to think it gets to design computers that face different constraints.
Can you share your crystal ball? I need to check next week lottery numbers
We have no idea what that limit is so it seems crazy to make any assumptions about where it is. At minimum it lies above Human Intelligence and even imagining what a civilization could do with millions of beings only as intelligent as say John von Neumann is frightening.
>those limits may fall short of magicing/smarting away the obstacles to colonizing space.
We can basically already colonize our Solar System with current tech. You don't need FTL or anything close to that to eventually colonize the entire Galaxy.
They may, or they may not, which is why I would not opine on the subject. I was only addressing one narrow part of the comment I replied to.
I'm not sure what's difficult about this. Are we not able to test what humans can run the fastest without being able to run just as fast? We can equally test which humans are the best spellers, or the best chess players, or the best problem solvers.
Once we move to more general forms of intelligence, like math and other forms of problem solving, then you apply more general tests.
For instance, if we had an Einstein AI v1.0 that could infer General Relativity using X0 bits of information in time T0, and Einstein AI v2.0 managed to infer GR using X1 << X0 bits of information and/or could infer it in time T1 << T0, then v2.0 is clearly superior to v1.0.
I agree that sometimes the parameters for such tests aren't always clear at first, but this has always been the case in science, which is we we iterate and progressively refine our tests as we learn. Any true AGI must be capable of such abductive and inductive reasoning to qualify as a general intelligence.
The amount of brain matter does not say all that much about the level of intelligence. Some kinds of birds are as intelligent as 7 year old children, and that's not something you'd guess from the size of their brains.
Your very intelligent AGI would most likely have no interest in seeding anything on Earth. It might reasonably be curious about our ecosystems, but disturbing those would reduce that interest.
Why do people keep acting as if this was a god given truth and just a matter of time ?
It's making the assumption that human brains are simply very complex neural nets. Even ignoring religious/spiritual arguments about souls, the simple fact of the matter is that we don't really know a lot about how the human brain/consciousness actually works on a fundamental level (and anyone who tells you otherwise is misinformed or lying).
How human brains work is irrelevant, we didn't need to solve the Navier Stokes Equation to build Airplanes and we don't need to solve intelligence either. Consciousness is totally outside the bounds of this discussion and also irrelevant to building an intelligent AI.
At what point do you think computer technology will reach a theoretical limit and cease to progress?
If they gave a specific time range, it would be.
Plenty of people have done so.
Indeed, we have AGI already, they are called humans, and the progress we see in the development of AI is not exponentially explosive. The only exponential growth we've seen (which is necessarily logarithmic growth after some inflection point, actually) is Moore's Law.
Actually it is. Human population growth has been exponential, as has been our resource use and consumption of our habitat, and the pace of our innovations. We've now created forms of machine intelligence to further augment our own, and machine intelligence could itself start advancing soon, without the messy constraints of evolution and biology. The possibility is definitely there, we'll see if it works out that way in the end. If it does, it will likely blindside us.
Is it? I'm not so sure about that. I'm pretty sure you can't be sure of that too. Given how machine learning continues to surprise us, I think this is one of those questions that will only be clear in hindsight.
People are too quick to forget how many "unassailable" towers have already been toppled. They see how kind of obvious each problem looks in hindsight, "oh it was just this one trick", not realizing the irony that yes, one or two tricks, and we may be only one more trick from being in serious trouble.
No they don't. They managed to get impressive results, but it's still far from coherent because the algorithms still lack the ability to think
Given that these papers have actually been published, the papers they produce are at least as coherent as the state of the art in the field.
> but it's still far from coherent because the algorithms still lack the ability to think
Nobody knows what "thinking" is, as a functional process. For all you know, human thinking may itself just be the same sort of pattern matching as used in machine learning, just with a few more parameters. In which case these algorithms actually are thinking, just with more limited intelligence than us.
But we can’t let self-driving cars injure or kill even 25% as often as human drivers do, so we refuse to use them to their maximum capability.
Keep in mind this thread’s premise is us developing an intelligence that can navigate space and maintain all requirements of that for millennia. Not driving a couple miles on a clear stretch of highway.
Your standard for "maybe" exceeds the abilities of many humans.
AFAICT, the statistics for fatal accidents do exist, and suggest that car AI may be marginally better than average humans — the caveats to that include at the very least (1) it's within the margin of error; (2) that includes drunks, dangerously tired, teenagers, and people like my mother and grandmother when their Alzheimer's was in the early stages; and (3) I don't trust the impartiality of the source.
But the stats do exist.
Also, many self-driving cars are not so much unsafe as occasionally bad at the basics of driving in certain scenarios - such as massively over/under-steering in intersections, leading to the need to maneuver in traffic (I've especially seen this in Tesla FSD videos).
Also,
> Your standard for "maybe" exceeds the abilities of many humans.
I'm very curious what human driver is simply unable to do that task. In fact, the vast majority of human drivers could do this task, but 0 current self-driving cars could even attempt it.
I've heard a lot of conflicting reports about this with regard to Tesla's AI, but regardless your point is sound, my list is not exhaustive.
> not so much unsafe as occasionally bad at the basics of driving in certain scenarios
Indeed. It's one of the counterintuitive things about machine intelligence in general, that it can be wildly superhuman in domains that humans consider challenging while also being utterly pathetic in domains we consider trivial.
> I'm very curious what human driver is simply unable to do that task. In fact, the vast majority of human drivers could do this task, but 0 current self-driving cars could even attempt it.
A quick google suggested that the roads in the alps are often closed in the winter by the police because of the snow, so hopefully in such conditions the answer is "100% of humans cannot do this" :)
My non-blind and somewhat functional 90-year old uncle can sit in a car and drive it quite legally at the moment.
I'm pretty sure self-driving cars are less dangerous on the roads than he is, but we're not willing to test that assumption.
I'm not sure what you mean by "machine intelligence," but there is absolutely no indication that Strong AI can be achieved let alone "pretty clear we're on a path," and compelling argument that it isn't even possible. Machines that learn, sure, but conscious machines, no way.
Your droids, we don't serve their kind here. They'll have to wait outside. We don't want them here.
There is no compelling argument or evidence that machine intelligence is impossible.
Physics tells us that machine intelligence is in fact very possible. All finite volumes contain finite information, per the Bekenstein bound. Therefore a human being can be fully described by finite information. Therefore a human being can be fully simulated by a Turing machine (technically just a finite state automaton).
The only way to escape this conclusion is to assert that humans are special in some way that can't be described by physics. You can believe that, but don't mistake that as "evidence" or a scientific argument that machine intelligence is impossible.
The current state of the art machine learning systems are still a few orders of magnitude away from the synaptic complexity of a human brains. A lot can happen over a few orders of magnitude, and we're closing that gap fast.
This is proof by assertion, but let's define what we're talking about first. I was referring to Strong AI. If you are also referring to Strong AI, then I can prove you wrong with not just one but two examples of compelling arguments against the possibility of Strong AI. Roger Penrose argues computers are unable to have intelligence because they are algorithmically deterministic systems. According to Penrose, rational processes of the mind are not completely algorithmic and therefore can not be duplicated by any sufficiently complex computer. Another compelling argument against the possibility of Strong AI is John Searle's Chinese Room thought experiment, and we know it is compelling for the Herculean amount of effort put into trying to prove it wrong for the last 40 years.
> Physics tells us that machine intelligence is in fact very possible.
This is more proof by assertion. You are asserting and not supporting your assertions in any way.
> All finite volumes contain finite information, per the Bekenstein bound. Therefore a human being can be fully described by finite information. Therefore a human being can be fully simulated by a Turing machine (technically just a finite state automaton).
It does not remotely follow from the Bekenstein bound that it is even humanly possible to perfectly describe a human, even if it sets an upper limit on the amount of information if it were possible, which it isn't, nor that perfectly describing a human will produce consciousness. A description is just that. A description of a book is not the book itself, a description of a human is not a human, and a description of consciousness is not consciousness, nor indeed could a description of consciousness itself be conscious. Further, the Bekenstein bound "implies that non-finite models such as Turing machines are not realizable as finite devices."[1]
> The only way to escape this conclusion...
More proof by assertion. You are asserting and not supporting your assertions, thus your argument is fallacious. Another way to escape your conclusion is to point out that you are misapplying the Bekenstein bound to falsely suggest that the limits it sets are or will ever be realized by humans, which they won't. We simply aren't capable and never will be capable of perfectly describing a human down to the quantum level.
> The current state of the art machine learning systems are still a few orders of magnitude away from the synaptic complexity of a human brains. A lot can happen over a few orders of magnitude, and we're closing that gap fast.
More proof by assertion. Assertions are not proof, nor are they valid argument.
Penrose is great but resting your argument on his work is dubious. I would go for a portfolio approach when thinking about these things.
> Proof by assertion re physics & intelligence
Physics does indeed tell us that intelligence is possible, inasmuch as we are intelligent. Unless you are claiming that you are not a materialist?
> More proof by assertion
The GP is just asserting that 'a lot can happen over a few orders of magnitude', they aren't trying to proove anything.
It seems like you are making your 'proof by assertion' rebuttal do a lot of work for you, without making and strong claims yourself other than 'GP didn't give good reasons why AGI will be a thing'.
Maybe GP didn't give satisfying reasons, but if you know so much about this stuff it would probably be a worthwhile thought experiment for you to attempt to steelman the Strong AI position instead of shouting it down.
As I was not resting my argument on his work, this is a straw man. I merely used Penrose as one of two examples to counter GGP's unsupported assertion.
> Physics does indeed tell us that intelligence is possible, inasmuch as we are intelligent. Unless you are claiming that you are not a materialist?
Physics does not tell us that strong artificial intelligence is possible in the way the GGP attempted to prove with invalid argument.
> The GP is just asserting that 'a lot can happen over a few orders of magnitude', they aren't trying to proove anything.
The use of words such as "thus," and "therefore" really gives it away that is precisely what they were attempting.
> It seems like you are making your 'proof by assertion' rebuttal do a lot of work for you, without making and strong claims yourself other than 'GP didn't give good reasons why AGI will be a thing'.
I am not required to entertain fallacious argument, and yet, generously, I identified the type of fallacy.
> Maybe GP didn't give satisfying reasons, but if you know so much about this stuff it would probably be a worthwhile thought experiment for you to attempt to steelman the Strong AI position instead of shouting it down.
Maybe I will. I just might do that.
That wasn't a proof, that was my thesis statement.
> If you are also referring to Strong AI, then I can prove you wrong with not just one but two examples of compelling arguments against the possibility of Strong AI. Roger Penrose argues computers are unable to have intelligence because they are algorithmically deterministic systems. According to Penrose, rational processes of the mind are not completely algorithmic and therefore can not be duplicated by any sufficiently complex computer.
That's not evidence, that's the assertion of one person who also has not provided any proof. There is no evidence that "rational processes of the mind" are not algorithmic. In fact, Solomonoff induction demonstrates that the scientific process actually is algorithmically describable.
> It does not remotely follow from the Bekenstein bound that it is even humanly possible to perfectly describe a human
Unless you are asserting that humans somehow go beyond the natural laws that govern the universe, which is what I mentioned as the only way to escape this conclusion, then this is not correct.
> Further, the Bekenstein bound "implies that non-finite models such as Turing machines are not realizable as finite devices."[1]
Not relevant. Firstly, as I said a human would actually be a finite state automaton. Secondly, not all Turing machines are physically realizable, but many are, and humans would necessarily be one of that latter class (the class expressible as finite state automata).
> A description is just that. A description of a book is not the book itself,
But if all you are interested in is the information the book contains, then yes, the description of the book is the book itself. Consciousness and intelligence are arguably information systems, which we observe by our ability to receive and respond intelligently to information.
If you're suggesting that consciousness or intelligence is something else that goes beyond information, then you are asserting something supernatural again, something beyond physics, and the burden is on you to justify this.
> More proof by assertion. You are asserting and not supporting your assertions, thus your argument is fallacious.
No, that's a conclusion that follows directly from preceding arguments, it's not an assertion.
> We simply aren't capable and never will be capable of perfectly describing a human down to the quantum level.
We don't need to do this. I'm not sure why you think this is remotely relevant. The only relevant question is whether whatever part of the human mind makes us intelligent can be replicated algorithmically. There is literally no physical evidence at this time that suggests that this would not be possible. All of known physics is computable.
Penrose claims, without evidence, that the mind utilizes quantum mechanics and also claims, without evidence, that this use of quantum mechanics is necessary for consciousness. However, in the end, quantum mechanics is just manipulating information, and classical computing can simulate quantum information processing by incurring only a polynomial slowdown, so this isn't really a barrier anyway.
> More proof by assertion. Assertions are not proof, nor are they valid argument.
You keep misusing the terms "proof" and "assertion". That statement was neither.
Your thesis asserted a false statement.
> That's not evidence, that's the assertion of one person who also has not provided any proof. There is no evidence that "rational processes of the mind" are not algorithmic. In fact, Solomonoff induction demonstrates that the scientific process actually is algorithmically describable.
It was evidence of at least two compelling arguments against the possibility of Strong AI, regardless of whether they are correct or not, they merely served to prove your "thesis statement" is false.
> Unless you are asserting that humans somehow go beyond the natural laws that govern the universe, which is what I mentioned as the only way to escape this conclusion, then this is not correct.
You've just claimed it is humanly possible to perfectly describe a human, while I'm pretty sure it's not possible for a human to perfectly do anything.
> Not relevant. Firstly, as I said a human would actually be a finite state automaton. Secondly, not all Turing machines are physically realizable, but many are, and humans would necessarily be one of that latter class (the class expressible as finite state automata).
You made it relevant by introducing the Bekenstein bound along with Turing machines in your argument. If the Bekenstein bound is correct then Turing machines are not realizable as finite devices. You can't have it both ways without contradiction.
> But if all you are interested in is the information the book contains, then yes, the description of the book is the book itself.
Unfortunately, we cannot redefine "description" whenever we like to suit your needs. The description of a book is not the information it contains, instead it not only describes the content but also the physical properties of the book, such as location, dimensions, weight, color, smell, etc.
> Consciousness and intelligence are arguably information systems, which we observe by our ability to receive and respond intelligently to information.
Whatever floats your boat.
> If you're suggesting that consciousness or intelligence is something else that goes beyond information, then you are asserting something supernatural again, something beyond physics, and the burden is on you to justify this.
Like nearly all academic philosophy of mind sources, I reserve the term "Strong AI" specifically to mean software and hardware combinations that experience sentience or consciousness. This is specifically what I have argued against, the possibility that this could ever occur, no matter how much time, no matter how many attempts.
> No, that's a conclusion that follows directly from preceding arguments, it's not an assertion.
Here's your fallacious proof:
1. All finite volumes contain finite information, per the Bekenstein bound.
2. Therefore a human being can be fully described by finite information.
3. Therefore a human being can be fully simulated by a Turing machine
>> We simply aren't capable and never will be capable of perfectly describing a human down to the quantum level.
> We don't need to do this. I'm not sure why you think this is remotely relevant.
Then your description can neither be full nor perfect.
> The only relevant question is whether whatever part of the human mind makes us intelligent can be replicated algorithmically.
The human brain has many parts, but we are not able to distinguish between any "parts" of "the human mind." Mind is both a faculty of brain and a phenomenon that arises from brain. Mind is neither a faculty nor a phenomenon of algorithms.
> There is literally no physical evidence at this time that suggests that this would not be possible. All of known physics is computable.
Again, infinite ignorance fallacy. It is evidential that algorithms are not brains.
> Penrose claims, without evidence, that the mind utilizes quantum mechanics and also claims, without evidence, that this use of quantum mechanics is necessary for consciousness. However, in the end, quantum mechanics is just manipulating information, and classical computing can simulate quantum information processing by incurring only a polynomial slowdown, so this isn't really a barrier anyway.
I merely used the arguments from Searle and Penrose as examples of compelling argument against the possibility of Strong AI to prove that your "thesis" was false. Further, the elements of Penrose's argument that I chose did not mention quantum mechanics. You have built a deeply confused straw man argument with sweeping generalizations and distinctly unsound reasoning.
> You keep misusing the terms "proof" and "assertion". That statement was neither.
"Proof by assertion" is an informal fallacy. I was identifying the fallacy that you committed over and over. You can claim whatever you like, but unless you support your claim, it is proof by assertion.
So what is the source of consciousness then? If it is something supernatural, like a soul, then yes reproducing consciousness in a machine is impossible.
But if it is ultimately just a biological process, no matter how complicated, then it could be simulated given true understanding and a fast enough/big enough computer?
I'm not sure there is a third possibility?
Healthy brain is the source of consciousness.
> But if it is ultimately just a biological process, no matter how complicated, then it could be simulated given true understanding and a fast enough/big enough computer?
Though commonly referred to as such in metaphor, computers are not brains. Computers also have hard limitations that are very different from the limitations of biological brains. Maybe it is possible to accurately simulate a healthy brain but is unattainable due to physical and economic limitations. Then again, maybe even if we exhausted the global economy and every natural resource to build a planet-sized computer and simulated every minuscule part of a human brain, every cell, every neuron, and all the electro-chemical interactions, and it still wouldn't be conscious, maybe that is a far better result than if it could be conscious. What are we even doing? I ask you, what is the goal and benefit here? How would that be better than the unconscious AI we already have and the automatons we can already produce now, but haven't? For a time, computers will get smaller, faster, more powerful, but if we haven't already achieved Strong AI with the largest fastest computers available today, in the future if room-sized clusters can fit in a cell phone, how can we expect them to be conscious? And if the room sized computers of the future can be made conscious, how will that benefit anyone?
> I'm not sure there is a third possibility?
Well, imagine having two separate conscious brains, the one in your head and another somewhere else that is also physically a part of you. Maybe we'll alter our species to evolve these separate brains, I just don't know what the point would be, as it would not at all be like multiprocessing or parallel computing, but more like slavery.
So what? To have implications for the Fermi paradox, we merely need machines that can replicate throughout the galaxy. Whether they are conscious or not is irrelevant.
Civilisations don't make choices or act. Individuals do.
Already there's plenty of humans who do all kinds of weird things. In an interstellar future there will be orders of magnitudes more beings and thanks to light lag, no means of asserting overarching central authority.
""" I'm not sure what you mean by "fleshy meatbag intelligence," but there is absolutely no indication that Strong Chemical Intelligence can be achieved let alone "pretty clear we're on a path," and compelling argument that it isn't even possible. Fleshy meatbags that learn, sure, but conscious fleshy meatbags, no way. """
Does organic chemistry have a magical property that machines cannot have?
I think (medium certainty) that because human minds were produced by evolution, which we can replicate trivially in simulations, that it is unlikely to be impossible for us to get there.
:)
I was using simulated evolution to directly reach an answer rather than caring about the agent (my fitness function was not agent intelligence); however, to go down the direction you're after here, back in my first job, one of my coworkers used simulated evolution to make agents in a video game, and I was never able to beat those agents in that game even on easy mode.
We have the proof that the I in AI can be achieved (us) and we have the proof that the A in AI can be achieved (our technology), so while there is no proof that AI can be achieved, I'm pretty sure there is nothing fundamental that prevent it
For example the way we build computers might be super archaic and the equivalent of the flint stone, a flint stone is nice but if you need a fusion reactor no amount of flint stones will do the trick
If we're trading compelling arguments, then there is one for consciousness not only being unnecessary, but also a drawback that evolution would happily optimize away given time, and basically not worth the energy it burns.
I don't have a handy list of papers to support this, but I can recommend two novels by Peter Watts, Blindsight and Echopraxia, both of which deal with this idea (apologies for sort-of spoiler), and which both do come with references to academic publications backing this and other ideas used in the books.
Considering we don't know what consciousness is or what it does functionally, any such argument is a fairy tale at best. It might contain some interesting insights or lessons, but don't mistake it for reality.
Dr. John Searle's objection is that software is not a mind, and goes on to explain why with his Chinese Room thought experiment.[1]
That's an argument for why we _shouldn't_ build them. But not an argument for why we _can't_ build them. Do you see the difference?
> So how are we supposed to make a machine conscious without knowing how we ourselves are conscious?
Evolution didn't 'know' about conscious either.
The pioneers of the Industrial Revolution had only the faintest ideas about thermodynamics, yet managed to build working steam engines. (And we improved them over time, as we learned more.)
For the 'Chinese Room' see eg https://www.scottaaronson.com/democritus/lec4.html or just have a look at good old Wikipedia: https://en.wikipedia.org/wiki/Chinese_room
Though I'm not sure why you brought up consciousness in the first place. It's irrelevant for the Fermi Paradox.
This is a straw man fallacy, because I was asked "What's your objection to "conscious" machines?", and my objection is that it is cruel.
> Evolution didn't 'know' about conscious either. The pioneers of the Industrial Revolution had only the faintest ideas about thermodynamics, yet managed to build working steam engines. (And we improved them over time, as we learned more.)
This is also a fallacious argument. Just because we don't know how evolution created consciousness doesn't mean that we can use our ignorance of how consciousness evolved to create machine consciousness. If you're only using this as evidence that consciousness is possible, the fallacy is begging the question, petitio principii.
Also, I believe there is here equivocation and conflation of artificial intelligence with natural intelligence. If we have the time, we can spend 600M years to evolve consciousness directly, but then it would not be artificial, it would not be machine intelligence, it would be natural or biological intelligence, and it would be a massive waste of time because it is already here and we are it.
> For the 'Chinese Room' see eg https://www.scottaaronson.com/democritus/lec4.html or just have a look at good old Wikipedia: https://en.wikipedia.org/wiki/Chinese_room
Do you really expect me to argue against your citations when you're too lazy even to summarize any particular counter-argument? It has been said that most of the work done in AI has been in trying to prove John Searle wrong, which suggests his argument is significant. A Turing Test does not prove consciousness nor even intelligence. All it proves is whether a human can be fooled. Arguments suggesting emergent properties between a human and a rule book are patently ridiculous, absurd on their face, because learning a book could not possibly ever make a book conscious. Where the Chinese Room thought experiment breaks down is there is a human involved, and most counter-arguments get tripped up here forgetting that it is metaphor and there is no human inside a computer where these types of counter-arguments would ever come true: no matter how much machine learning you submit your program to, there is no mind to begin with that could ever become aware.
> Though I'm not sure why you brought up consciousness in the first place. It's irrelevant for the Fermi Paradox.
In fact, I did not, the OP of the tread asserted: "Seems pretty clear that we're on a path to machine intelligence," and I responded to that specifically and argued against the possibility.
OK, you want to argue both against the _should_ and the _could_, I guess?
About the Chinese Room: sure I can replay the standard arguments, if you want me to.
But if you already reject the validity of the Turing test, there's not much to discuss on that front, I guess.
How do you know that you aren't the only conscious being in existence? How do you know whether other commenters on HN are conscious?
> It has been said that most of the work done in AI has been in trying to prove John Searle wrong, which suggests his argument is significant.
Who said that? Lots of other things have been said, too. That doesn't suggest anything.
> Where the Chinese Room thought experiment breaks down is there is a human involved, and most counter-arguments get tripped up here forgetting that it is metaphor and there is no human inside a computer where these types of counter-arguments would ever come true: no matter how much machine learning you submit your program to, there is no mind to begin with that could ever become aware.
Aren't you assuming what you want to prove here?
Btw, would your argument still apply in this form?:
> [...] there is a human involved, and most counter-arguments get tripped up here forgetting that it is metaphor and there is no human inside a [brain] where these types of counter-arguments would ever come true: no matter how much [of anything] you submit [a brain] to, there is no mind to begin with that could ever become aware.
OK...? Strong AI is not possible for reasons given. Strong AI, if possible, is slavery, thus unethical.
> About the Chinese Room: sure I can replay the standard arguments, if you want me to. But if you already reject the validity of the Turing test, there's not much to discuss on that front, I guess.
Straw man, I have not rejected the validity of the Turing test, merely making the observation that a program passing a Turing test is merely fooling a human into uncertainty whether they are interacting with a human or a program. The Turing test itself is not proof of consciousness, it is only proof of a human being fooled.
> How do you know that you aren't the only conscious being in existence? How do you know whether other commenters on HN are conscious?
Solipsism and problem of other minds, and as such, a trap. Knock yourself out: https://plato.stanford.edu/entries/other-minds/
> Aren't you assuming what you want to prove here?
Nope.
> Btw, would your argument still apply in this form?: > [...] there is a human involved, and most counter-arguments get tripped up here forgetting that it is metaphor and there is no human inside a [brain] where these types of counter-arguments would ever come true: no matter how much [of anything] you submit [a brain] to, there is no mind to begin with that could ever become aware.
No, this is building a straw man and sounds a lot like homunculus theory.
Either English is not your first language, or you are just playing stupid games to avoid answering. or maybe both.
You were clearly not asked if you though it was immoral, you were clearly asked why you think it's impossible.
I honestly don't think that anyone would make that specific argument. Do you think that they are? Who's suggesting that we "use ignorance to create consciousness"? It seems an odd diversion to take.
> If you're only using this as ( consciousness has evolved) evidence that consciousness is possible,
Is it evidence that consciousness is possible? I think it is.
Is it evidence that consciousness is possible, from only ordinary matter and energy and no divine spark? I think it is.
Is that an argument that a constructed machine consciousness is also not ruled out? I think it is.
> the fallacy is begging the question, petitio principii.
I don't know what you're getting at here, or why.
I always found that experiment to be extremely unconvincing, to put it mildly. Using that framework I can prove that you are not conscious.
I construct a "english room". You are in the back without any contact to the outside world. A Chinese speaker in the front can pass messages to you from the outside world, in English by laborious copying. Clearly you aren't conscious because the Chinese speaker can't understand the English.
Or, alternatively, no one would dispute a single neuron is not conscious and can't understand english. I pass signals into your neurons and through some electrochemical magic you reply in english, yet the neurons involved don't understand english therefor you aren't conscious.
All Searle has done is picked out one component of a composite system, concluded that in isolation it has no clue whats going on, and from that concluded the system as a whole doesn't.
We have a hard time clearly defining consciousness, and it's not clear if some AGI would be conscious (maybe it's just an emergent phenomena)-- it's also not clear if there would be any particular advantage in it being so. But in any case, I don't think the chinese room is a useful argument for anything.
Our own brains are an excellent proof of concept that machines can be made to think. We're all machines in a way, after all - unless you believe there is a special supernatural exception for human minds.
A machine is an artifact. It is a human-built assembly of materials that have no natural inclination to work in unison. Silicon and copper, plastic and glass have no natural propensity to compute. We make artifacts, for example computers, out of these materials to serve our purposes. But they are not the materials’ natural purposes.
In the classical hylemorphic (Aristotelian) perspective, an artifact is an artificial composite of substances and thus it has an accidental, not substantial, form. Machines work as they do “accidentally”; that is, their function as machines is not inherent to the matter that comprises them. Rather their function is imposed on the disparate parts by human intelligence.
Unlike a machine, the brain is an organ, a functional part of a living organism. It (along with the body) has a substantial form; its activity is natural to it. One might say that the brain's activity is intrinsic to it, not imposed on it extrinsically, as is the activity of a machine.[1]
[1] https://mindmatters.ai/2018/10/yes-your-brain-is-a-machine-i...
So we looked and saw that life is just molecular nanotech, but what we didn't see was any tags attached to carbon chains that would say, "this molecule's natural function is to think", or "that molecule's natural function is to be a part of an organ that pumps blood", etc. Because there is no such thing like a "natural function" in the sense used by your quote, the distinction between "natural" and "artificial" is itself artificial[0], and that whole quote represents a piece of magical thinking philosophy you'd think humanity abandoned at least a century ago.
----
You could just let the discourse end there - it's a strongly held belief and that's the end of the discussion. What confuses me, and maybe you can clear this up in a reply, that you still want to have a discussion about facts and reasoning. For that to happen, you would need to show that "nature" does indeed have the privileged and unreplicable position you assert it does. Just using the term "nature" is rhetorically evocative but not rationally meaningful in this context. To start with, you'd also have to definitionally delineate nature from artifice, because we humans have really been blurring those lines for a long time now and no cosmic force has stopped us yet.
Your implication that "natural" processes can never be functionally replicated or continued upon by volitional development is not supported by evidence. There is, however, a lot of evidence for the other side of this debate. We are heavily altering "nature" to suit us, and wherever we looked so far we find machinery that can be rationally understood, manipulated, replicated, remixed, and used whole or in part in our own designs. If there is a supernatural component that somehow privileges "natural" life, we have not found it yet.
and the rest of your paragraph is about artificial vs natural, which isn't even relevant, even if it could be clearly defined. Asserting "Proof by assertion is fallacious argument" is a fallacious argument.
I don't think that you understand parent comment's very valid point.
If the question is "Can genuine (human-level) consciousness be constructed out of nothing but atoms and energy?" then the only possible answers are:
1 yes, and at human scale, since a human being is a worked example of such, made from nothing but atoms and energy.
2 no, a human being is not just made of matter and energy, there is some "extra" special part to a human, call it soul, spirit, divine spark, etc without which there is not "genuine consciousness". (a)
And the whole rational project of science, going back 100s years, dismisses option 2 as sheer useless mystical mumbo-jumbo with no explanatory power at all. Therefor, option 1 is valid.
a)
If we replicated a human brain cell by cell wouldn’t that thing be conscious? Maybe we’d have to replicate a gut for it too, as a secondary brain, but then wouldn’t that thing be conscious?
And if we started replacing parts of it with plastic, transistors and CPUs wouldn’t that thing still be conscious
If we copied a human brain, assuming it were possible, what we'd be left with is two human brains. But I doubt either could be conscious, but even if they both were, they're still human brains, not computers. The mistake is confusing artificial intelligence with human intelligence, and the exercise would be like something out of a Mary Shelley novel and not computer science.
The question then is "how powerful a computer", and nobody knows that, because even the lower bound estimates are too expensive to bother with at present.
If consciousness is a substance, and not an emergent feature of data that just happens to be processed in us via the medium of electrochemical pathways pumping protons across cell walls, then it is natural to ask if this substance remains after death[0], or precedes birth, or passes between creatures (not necessarily all human) in a cycle of reincarnation[1], or even between living and non-living things like rocks and rivers[2].
Nobody has found significant evidence of such a substance, though they have been looking.
[0] Most religions
[1] Buddhism, Hinduism, Wicca
[2] Some forms of animism
A simulation of two entities swapping data and creating a new mixed third entity is reproduction.
It’s feasible that a simulation of a brain would be a brain.
With "ifs and buts" we could go very far
Without Ifs, for example, Einstein couldn't have come up with anything of note.
It's also worth noting that the types of star or planet available don't matter much to meatbags either once they have learned to leave their homeworld. Orbital or even deep space habitats are much more efficient and easier to construct than livable planets. As long as there is a reasonable energy source nearby, you can have any environment you like in a spinning cylinder.
> and that interstellar voyages will be undertaken by those intelligences
Probably, although I would much prefer if we humans became hybrids of some sort (maybe uploaded and virtualized, or extensively augmented) instead of just falling by the wayside. It would be sad if our only job in cosmic history turns out to be giving birth to the real main characters of the story.
Well, would you consider yourself to be a heavily augmented fish? (Or heavily augmented amoeba?)
If yes, you can probably re-interpret our successors in that light, too.
Among other things, yes. There are enormous amounts of biochemical machinery we are still running today originating from the dawn of life. Although I believe intentionally designed augmentations would be (and in a way: already are) a complete game changer, in at least as much as the advent of higher reasoning and shared culture was the game changer that gave birth to us. So there may well be an insurmountable wall of separation between us and our successors.
From my admittedly biased perspective I think the most important aspects worth porting over to a new species are not even necessarily biological, but cultural and emotional. I personally think there is a potential for experiential richness that I would love to see explored in a human-like but highly advanced lifeform. To be clear, I wouldn't object to personally being that lifeform, either, but alas I was born too early.
The things being created and reproducing will still be humanity, just not meat-based humanity.
The machine intelligence - spread solely over the airwaves to receptive civilizations, which then get taken over to promulgate further spread. Laser launch telescopes have great bandwidth and transmit at approximately the speed of light, but require some selectivity.
The intelligent machine - Robotic Von Neumann probes, probably of the star-wisp design. Can colonize barren rocks, and be sent out in large batches at low speed.
The intelligent race - The fleshy bits. Not even very interested in barren rocks compared to planets. So logistically difficult that the supporting infrastructure limits them to tiny fractions of the speed of light at the cost of large fractions of civilizational output.
A suitably advanced civilization might be able to genetically re-engineer their bodies with live longer, and in more exotic environments.
They may also go the cyborg route and have mechanical appendages to their bodies, rather than replace their bodies entirely.
They may also be able to do things like put their bodies in a state of hibernation during interstellar travel.
The energy requirements for interstellar travel are so vast that the life support requirements for thousands of meatbags for possibly thousands of years really are just a rounding error.
Minimum possible life support for a human ≈ human basic metabolic rate ≈ 73 W
Divide energy by power to get time; that's 20 million years, which at that speed would get you to Andromeda before the lowest possible energy cost of life support reached the lowest possible energy cost of the velocity.
Humans are already a hybrid organism that works to ensure survival not only of genes but of culture and knowledge. It's not difficult to imagine a machine civilization would do the same: not only preserve their construction patterns but also their intellectual achievements.
However, none of that is strictly necessary - because evolutionary pressure still applies to machine species in the same way as it applies to us. Machines that stay at home and don't propagate, well, won't propagate. Eventually someone comes along who does propagate and the stagnant species will fade away. There isn't really any difference in mechanism between what we consider biological lifeforms compared to machine ones.
Edit @jimmytucson: HN doesn't let me respond directly, so I'm clarifying my answer here:
> And I can’t see any reason why it would, tbh
It doesn't technically need a reason. The fact that those machines who do replicate eventually dominate the landscape is enough, on its own. If 100 billion machine civilizations choose not to procreate but one does - that's the one that eventually seeds the ecosystem.
> There isn't really any difference in mechanism between what we consider biological lifeforms compared to machine ones.
There is one important difference: we can make machines with vastly better error correction for their replication. That has large impacts on any 'machine evolution'.
Your argument still applies. But there might not be any noticeable change in the machines long after the stars burn out.
Especially if initially they are engineered to have a long generation time. And if you take into account that most mutations are bad for fitness.
Absolutely, I think we agree on that. It's fun to think about whether machines with some intentionally elevated drift rate may occasionally outperform locked-down ones, but I doubt it would be worth it since I don't see a random walk algorithm like evolution beating intentional design strategies implemented by extreme intellects. (Especially since they'd be able to simulate designs beforehand and could easily consider vast numbers of variations in advance...)
Naively, one might suggest frequent AI uprisings as a solution to the Fermi Paradox. But on closer inspection this falls apart: even if the AIs take over, there's no reason to believe that they would be any less expansionist than what came before. So an AI uprising wouldn't make us see any fewer alien civilisations.
However, suppose we accidentally created grey goo and it eats the planet. The grey good might have low enough mutation rates, that if it hasn't been created with advanced capabilities in the first place, it will never venture out to the stars. It will just stay on the planet, unchanging until the stars burn out.
Those are my thoughts as well: for the purpose of the Fermi Paradox, it doesn't really matter whether it's machines or not - it doesn't even matter if they're "conscious", they could be the equivalent of bacteria colonizing everything.
I think you nailed it on the head when you said their drift rate largely determines their future. I consider beings with (for lack of a better term) high general intelligence to be innately high drift organisms, whether their actual replication processes are locked down or not. The drift they experience will be guided by volition.
Whereas machine bacteria (which we are) would be subject to traditional evolutionary processes. I'm really uncertain about the prospects of locking down the latter with high error correction, because it takes only one such bacterium with a random mutation to escape the error correction mechanism and voilá we're off to the races.
It's worth noting that organisms on Earth have a high variability when it comes to their mutation rate, which suggests the mutation rate itself is a parameter guided by a fitness function rather than an inherent byproduct of our specific biological substrate. Of course I agree with your intuition that much higher error correction rates can probably be achieved in architectures other than DNA/RNA.
In the end this is mostly a meditation on how static life in the universe will end up being. Not just regarding the time frames but also the depth of change. On balance, I suspect we'll end up with a highly variable ecosystem just by virtue of more pathways leading to that future.
That is my source for concern, because any such program would have to find self-replication to be some universal “good”, and not just a property of certain dynamical systems like viruses, bacteria, and eukaryotes. And I can’t see any reason why it would, tbh.