As described in the chinese room thought experiment: https://en.wikipedia.org/wiki/Chinese_room
Answers to the question will probably come from the convergence of two fronts: understanding better what the brain does, and correlating that with an understanding of what is going on in the machine. That is, maybe it will be shown that the kinds of states and state changes in the two are very similar (or can be mapped to each other in some way). In other words, we externalize the structures and states as much as possible and identify an equivalence. Then we can proclaim that what is going on in the machine isn't just some elaborate script; the states are actually human like: the external behavior is underpinned by apparently the same stuff that makes people tick.
We will also simply be convinced on an emotional level, due to the machines leading complex lives, demonstrating traits such as shame, regret and self-loathing, as well as joy that looks genuine. There will be depressed AI's that require therapy (possibly from humans), and ones that choose to terminate themselves. People who are not computer scientists or philosophers will be convinced that the machines are conscious, by the ways in which their lives intertwine with those of the machines and the relationships they form; only a dwindling group of skeptics will remain, even long after there is no space on the field where the goal-posts can be moved any farther.
Who will decide how many of these AI's can be created? Will AI's ultimately be able to create their own offspring? We already have limited resources, this will only increase the demand :-) I see trouble ahead... as well huge opportunities for advancing our understanding of the Universe.
What's not clear to me is that, in order for us to ascribe AIs the moral value of consciousness, there has to be an isomorphism between the digital computer's computations that generate their external human-like behavior and our brain's biological computations that generate our external human-like behavior.
For one, even if there is such a mapping, there may be difficulty in expressing it in ways our minds can understand. (Side note: would we have consult with the seemingly-sentient, newly created AIs to act as proof assistants to decide whether we humans should ascribe moral value to them?) So inability to for humans to confirm the existence of such a mapping isn't really that indicative of anything.
For two, more fundamentally, suppose that parts of the AI consciousness algorithm do map to parts of the human consciousness algorithm, but other parts don't. How do we figure out which are "key" to the One True Consciousness? AFAICT even in theory the only way to figure that out would be for an individual human to sign up for exotic (and impossible) surgeries to radically modify how their brain works to emulate the parts of digital computations that don't map to biological ones.
Honestly, it seems almost bigoted to me to say there's only one way for consciousness to exist, and that's our own. Biological consciousness supremacy.
The only good part about this thought experiment is that it makes the hard question of consciousness - how can it arise from physical phenomena - somewhat more obvious. But it's still the same question as for normal human minds and neither an argument for software systems becoming conscious nor against it.
(BTW: This is not just meant as a joke, nor just as instigating fear of the singularity - turning a question on its head can sometimes lead to interesting insights).
Just to be clear, I hold no solid viewpoint on this, I'm just interested as to what other folk here think.
I'm with you though: consciousness and sense of "self" are just matters of complexity of the brain.
Although, it's possible that creating something that does what humans do requires the same materials humans are made of (but seems unlikely, or at least I think other materials will allow a close approximation).
Unless you belief in a mystical soul component (I do not), there is nothing else "there" but machine. A very complex, biological machine. But there is no reason to believe that that a sufficiently powerful software/hardware can not replicate the functionality of that machine exactly. Then what possible difference could there be between the two duplicate machines?
But, we will have systems that exhibit a many aspects of "thought" and sentience long before we are able to code up homo sapien sapien.
https://www.quora.com/Is-the-human-brain-analog-or-digital
The development of quantum computers could offer a huge leap forward with the development of AI.
But will people accept this?
Let's start by looking at how brains work for something that is much easier to reason about - vision. Light hits the eye, and it activates a set of neurons that are wired up to the light receptors. There's further out neurons that are set up to catch certain higher-level patterns, like edge detection or objects headed directly toward the eye. Deeper and deeper these connections go, until you get something that you subjectively experience as "seeing". It's at this point that optical illusions work - flat images on paper that are designed to make the higher level visual pattern recognizers fire. At a very high level, how parts of the brain work is by activating based off the sensory data represented by the activation of other parts of the brain.
Just like detecting edges in your visual field was important in the ancestral environment, so was answering questions like "what did you do on the hunt?" with a good story. Just like there's a portion of the brain the constructs edges from lower-level visual sensation, there's a portion of the brain that constructs narratives from lots of different kinds of sensations. Furthermore, this narrative module has access to remembered narratives (since consistent stories are better ones) as another form of sense data.
That, in short, is what consciousness is - the brain making sensible stories out of the sense data it encounters, and making those stories available to the brain (including the story-making module, which is why you can make stories about the qualitative experience of story-making).
Simply declaring 'that's what subjective experience is' is not an explanation.
Another way of putting this is that you haven't explained why it's necessary that an autobiographical message passing system needs to experience itself, and what the threshold is for qualifying as such a system.
In other words, that representation is privileged because the environment often demands story-telling from you. In situations where it doesn't, you get very different experiences (like flow states, or long solo wilderness hikes).
2. What is the difference between "having" consciousness and "simulating" it?
Awareness, memory, leaning, anticipation and subjective experience.
Empathy is a good example, will a computer ever "feel" empathy? Or will it be simulated empathy? Is there actually a distinction? I'm not sure.
For instance, is Android a Linux distro? Depends why you're using the phrase "Linux distro". For some applications of that phrase, yes; for some applications no. Saying that it is, or that it isn't, doesn't tell you anything more, and doesn't help you answer those questions; you still have to answer them, so you might as well answer them directly. For instance, if you're packaging some software for a bunch of Linux distros, you probably don't want to count Android (or at least, handle it very differently from Debian or Fedora). If you're tracking fixes to kernel CVEs, you probably do want to count Android.
Or, more mathematically, is 0^0 = 0 or 1? Depends how you ended up with 0^0 in the first place: if through 0^y as y approaches 0, then 0, but if through x^0 as x approaches 0, then 1. Saying "It's 1" or "It's undefined" is satisfying to the part of the human brain that likes clean answers, but it doesn't inform us.
Similarly, let's not ask "what does 'consciousness' really mean", but "where do we use the term 'consciousness'." Should AIs be able to drive a car unsupervised? Should a sufficiently advanced AI have civil rights, such as the right to life? Should an AI be entrusted with political office? Or with the vote, and if so, what representation is fair? Can I give an AI ownership of a company? For the religious among us, do AIs have the ability to have human-like morality? Those are all questions worth answering, but saying "Yes, this computer has a conscious mind" or "No, it doesn't" is not really going to help those questions get answered.
The Chinese room experiment is sort of not useful in that, while "the ability to learn Chinese" is a thing we expect of consciousnesses, it's not really generalizable. Nothing else in our society really depends on people's ability to learn a language; those few things that do, generally don't require the learner to be a person (e.g., you want something translated and you give it to a company).
In this case, all that's happened is that a game played by people can now be played well by a computer program which is designed specifically to recognize patterns. (A go position is just a 19x19 array of trits.) Computer programs reached human-level performance decades ago at chequers/draughts, and later at chess and backgammon. Those programs did not lead to a general AI.
Will we construct machines with minds despite that? Since "to see if we can" is usually sufficient motivation for humans, I expect yes.
Biologically-inspired computing techniques will probably lead to computers with minds, and I can see these computers eventually becoming smarter than humans. Will we make such machines before we invent enough of future-statistics to make a mindless machine that's smarter than us? I don't know, that's an interesting question.
Personally? Yeah, I do think computers will eventually achieve "consciousness". I don't think of consciousness as being anything magical or terribly special to the point that a computer couldn't have it. I generally think it's just a question of a sort of recursion where the brain is "hearing" it's own internal processing as it works. When we ask questions, run simulations (day-dream) and we "see" and "hear" those things, I suspect it's just some sort of feedback loop between internal modules of the brain.
I'd go so far as to suggest that some computer programs may already be conscious, and we just don't have any good way to measure/test that fact.
Because an airplane doesn't flap it's wings to fly like a bird does, should we say it isn't really flying?
The link you provided seems to indicate that people with reasonable credentials, either in Philosophy, or AI, struggle with it.
It's likely that eventually the boundaries between humans and AI will blur. I guess at some point there will be cognitive augmentation, but imagine the debate that will generate. Look how we view physical enhancement by drugs today in sport.