Right. But that "software brain" will be completely deliberate.
> subjected to all kinds of experiences, which, in turn, will result in this brain becoming conscious - gradually.
Again, there is no evidence that "all kinds of experiences" will result in gradual emergence of consciousness from anything we have observed.
To a computer, experience is just data. So we're not talking about computers.
Phrases like "software brain" and "conscious machines" are ways of adding something that is not software to computers. But trust me, if we're talking about software, the programmer will know every line of their code.
Now, if you want to get meta, we can consider programs that write programs. But again, the first program will be written by a human, and we will know exactly what we are doing.
> resulting inferences more or less defy rigorous explanation
We have written programs that take input and produce data models. There is no consciousness here.
Humans take input and produce models too. But I fear you may be confusing learning with consciousness and agency. These not to be found in the code of learning machines, nor in the models they've learned. I am sorry, but it's just not there.
For every abstraction there is an implementation. That is physics. With no implementation, we have metaphysics. Here, "consciousness" is the abstraction. The physics of it is the physical implementation, and is what Roger Penrose and other scientists are working on identifying. The simulation will have a coded implementation. Meaning, whether we generate it or we build something that generates it, when we have "consciousness" we will have the code that backs it. To be able to do this without any understanding of what consciousness is would be analogous to monkeys typing Shakespeare.