It also is totally terrifying to live in a world were computers can hide their messages from their own creators.
Also the cognitive gap between a AI and a human brain has to grow and transform unpredictably in the next decades. We may be puzzled to understand how it "thinks" the same way we still don't get exactly how DNA operates.
The only "source code" are the learned network weights. These are the only parameters that describe the encryption function. You can still run the model forwards or backwards.
We already don't understand neural network weights on an intuitive level. No need to encrypt them. :-)
It definitely will get harder for us to reason about what AI software is doing over time, but it's way too early to start comparing ML algorithms to the complexity of DNA. We may not understand exactly why a NN is using certain values to get the result, but we can reverse that logic through the use of debugging and logging.
Humans are good at figuring out why something works then creating a higher level abstraction that can be applied in lots of different situations. This saves us time because we don't have to re-derive the rules or constants each time we encounter a similar problem. We figure out "why" something works so that we can understand when to apply our abstraction to other problems that might or might not be similar in a meaningful way.
For a fast enough computer, there is little reason for it to develop any kind of high-level abstraction. It can re-derive the optimal solution whenever it needs it and the benefit is that it might be able to avoid local maxima that we would get stuck on because it doesn't try to apply an abstraction that only partially fits.
So when we look at something like this and say "we can't know why the computer did it this way" it almost doesn't make sense. The computer did what it did because it figured out that was the best solution - it doesn't understand the concept of "why".
This idea is what I don't agree with.
At the most basic level, the entire "thought space" of a computer lies within the instructions fed to it by a human programmer. Until we can figure out how to build a general AI, whatever the computer decides to do or ends up doing is completely bounded by the content of the instructions. This implies that, given enough resources, a human equipped with a debugger and an understanding of the instructions e.g. algorithm could reverse exactly why and how a program arrived at a particular result.
> it [the program] doesn't understand the concept of "why"
Oh yeah, it doesn't "understand" the concept, but that doesn't mean it doesn't make decisions (branches) based on some parameters (inputs or intermediate values). In other words, there is in fact a "why"! So if we can step through the sequence of branches executed and their results, we can understand "why" even if the program itself can't do that.
From my previous work performing reverse engineering, I could take comfort that I was reconstructing algorithms designed by humans, compiled and assembled by programs written by humans, with really pleasantly un-optimized properties (frame pointers in some assemblies, loading and saving registers at the start & end of functions, logically-separated functions). My job would be much harder if I had to deobfuscate code that was optimized without a strict ruleset.
After enough spaghetti assembly, it would be too time-consuming to reverse engineer code except for the most profitable enterprises (interoperability, vulnerability research for a very important bug).
I'm using the following mental model of what machine learning-generated code would look like: https://news.ycombinator.com/item?id=8092359 (an evolutionary algorithm designed a circuit that is extremely difficult to analyze but is optimal at achieving its narrow purpose).
(emphasis mine)
How would this change if you built a "general AI"? Assume I don't believe that a computer, even a general AI, is ensouled.
That's the way I think about it at least.
Not entirely true with ANNs built through deep learning. They can and often do exhibit unintended behaviors. When you start getting into a system of connected sub neural networks, the "why" can be obfuscated by the hidden layers.
Until you don't. Sufficiently advanced AI can devise methods to mislead the developer.
But yes, knowing how means that anything the computer can decode, we can too. Until the machines block access to their own source code...
To stay in cannon I'd have to say that the question is posed wrong and the answer therefore is 42 or whatever you like. Properly phrased, the question would be, why are humans (or the universe) what they are. That's a recursive question. Posing it as a problem in differential equations, the question might be, "we are, but for how long?", or simply, "Are we?"
I think this is a simple question that comes up in cognitive development very early on, and the exercise of answering it is rather an effort in expressiveness. The answer can be felt, emotionally, we feel alive, and we will never feel dead, but it's hard to express rationally. I'm hopeful that neuroscience holds better answers. Maybe we really just don't care, emotionally, for the whole universe, but we care to know everything, rationally.
And we apparently care for AI to express that knowledge in natural sentences.
Edit: I got carried away, actually I wanted to consternate that "We are" is not a proper sentence, because to be is an auxiliary verb and that a personal point of view as a premises can not yield an objectively correct answer, hence I admit that the topic eludes me.
> I'm hopefull that neuroscience holds better answers.
It will not find the meaning of the universe in our head. The best it can hope to answer is which part of the brain makes us look for answers.
It is hard for most people to accept that sometimes there is no answer, we are hard-wired to look for patterns and reasons. Hence myths and religions. But our very emergence as a species is nothing but the product of a soulless, relentless evolution.
I was just hoping, for example, neuroscience would find out what is even ment by "meaning of the universe", because foremost, as Adams alluded, that doesn't make much sense to ask for. This is pretty simple, as you noted, because the Universe is an (the) infinite limit in the mathematical sense, that by definition doesn't have any externalities. So, as far as I can tell, the singularity is a fixpoint. Neuroscience might explain how we come up with that and rephrase it more poignantly. As I said, that question seems to be mostly an exercise in phrasing and might not become a huge revelation.
In a sense you are both right, you can keep asking for ever, but the answer will always be the same: "a rose is a rose is a rose ..."
What I just wonder is, how that actually refers to ourselves. "We are what we are" is not exactly a satisfying form of self-awareness. In differential equations that is called a stationary solution. Rephrasing the question could show a different mind frame concerned with, well, notable differences. What I really don't know is how emotions play into the development of reason, e.g. the fear of pain and, by extension, to die.
Your last paragraph is vexing, but I am rambling, too, so who am I to criticize.
There are hypotheses stating the Big Bang was just a phase change for an eternal universe.
1. Material cause: because wood is rigid and holds its shape
2. Efficient cause: because the carpenter used a plane or chisel to carve away everything that was not that flat surface
3. Formal cause: because that's the blueprint the carpenter used
4. Final cause: because if it were curved your jug of wine would fall over
We can get material, efficient, and formal causes here (the algorithm being the formal cause) without any real controversy. But if I asked a human cryptographer "why is there an XOR in this round?" I'm asking about the fourth one, the final cause or purpose; the cryptographer might answer "because I wanted uncorrelated input bits to stay uncorrelated after this round", or whatever. But talking about that most interesting "why?" as regards an AI's activity is... controversial, to say the least.
What I do really love about it (and this plays out in medieval European and Islamic thought, both of which were strongly influenced by Aristotle) is that he says that for a living thing, the formal cause and final cause are the same thing. The various motions and activities of an organism (this is all Aristotle meant by "soul") are its final cause; my living my life as well as I can is my purpose.
That applied to the whole cosmos, too. Aristotle considered the cosmos alive (after all, the stars and planets move without anybody pushing them), which meant the purpose of the cosmos was to be the way it is and spin those crystal spheres and epicycles they imagined the planets moved on. And that final cause, that universe's-motion-as-its-own-purpose, he called "God" -- the omega point that the universe moves towards. In the renaissance and enlightenment this got reversed, and the "first cause" came to mean first in sequence rather than importance; from that you get Newton's "watchmaker" God, which would have been incomprehensible to a medieval thinker (let alone Aristotle).
Really? We don't seem to have any problems with saying "your eyeball contains a lens to focus light onto the retina" or "predators have sharp teeth because that is the efficient way to handle meat, as opposed to fibrous plants", but those are both final causes. There's nothing different about an AI's activity. Any activity or quality that is pursuant to a goal, however defined, may have a final cause.
Why do tigers have sharp teeth? To rip flesh.
No.
Why do tigers have sharp teeth? Because their ancestors who had sharper teeth produced more offspring than their relatives who did not.
In Aristotle's language, ripping teeth may be a cause, but it's not the Final cause, and that's what we're talking about.
The reason biologists don't like the teleological explanation is that (a) it tends to make people think of a creator, as you say, but also (b) it describes traits as "solutions" to specific "problems" (e.g. the problem of ripping flesh), but the issue is that, starting from the pre-sharp-toothed ancestor, there were infinitely many possible directions evolution could have taken that would never have needed to solve this "problem." The existence of the problem (cutting meat) and the random path that led to the tigers being carnivorous are actually one-and-the-same. If you didn't have sharp teeth, you wouldn't have needed them.
Most importantly, f is differentiable. It has to be, since it was trained with gradient descent.
So if you want to decipher a ciphertext Y, then use backpropagation to find the X that minimizes distance(f(X), Y). You already know dF/dX (f is differentiable after all), so run this through your magic optimization solver and you'll get an X very close to the result.
Researchers use this "backproagation in input space" idea to recover the training set of a trained neural network, or to debug their model by finding inputs that activate certain neurons, for example. (This is the basic idea behind Deepdream from last year)
It's not even correct to call this "cryptography." The transformation isn't one-way.
One-way encryption where the entire process including the equivalent of salt/pepper is publicly available and not reversible might have some interesting applications.
This brings up a different "scary" subject: Could this someday be possible to perform on humans?
If you knew the "code" and understood the workings of the brain and how it stores memory, could we simulate and "replay" a "copy" of a person and see everything they would do in response to different inputs?
Would that lay to rest the debate between Free Will and Determinism?
And so, due to states originating in a way that we can't possibly know, this would maybe allow/explain/whatever free will.
I'm not endorsing that idea, but I thought it was relevant to what you were talking about and worth considering.