Train on plaintext, ciphertext -> key.
More classically, you can try feeding the problem into a SAT solver. People have tried that too. Doesn't work - it just grinds until you run out of memory or patience, finding no useful results.
You can also try doing it by hand and see if you get anywhere (you won't). People have tried.
This is an adversarial problem. The problem is literally designed to be resistant to all kinds of analysis. That's the point. Even real attacks, like SHAttered (different kinds of attack on a different kind of algorithm) manage to find conditions where the probability of finding a solution is raised to 2^-70 or so, and then they let it grind on their biggest compute clusters until they find one. And that problem (finding a collision in a cryptographic hash function) is one that's especially amenable to grinding. If you're mounting a known plaintext attack it's unlikely your adversary will answer 2^70 encryption requests for you.
Every encryption algorithm proposal has this property of being designed to have no patterns....
And, just because what I'm saying isn't especially likely to work, it's not obvious that it cannot. Very large models are doing all manner of things that very smart people thought were not possible just 6 or 7 years ago.
If you were in a place to debate this, you would have known the above (or something similar) is what I was suggesting when i said train on plaintext, cipertext -> key, and you'd have some deep mathematical insight as to why no architecture known is likely to work. And you would also know I wouldn't be here talking to you about it if I really had a solid idea of an architecture that is likely to work.
I think it would make sense to explain how a theoretical model could do better than SAT. Otherwise, is the idea here just "magic is possible"?
Current SOTA language and vision models, or models used to predict protein shapes are magic by the standards of 2016. As for why could it be better than a SAT? Why couldn't it be? Models are better than deterministic, logically written software for lots of situations. You can create infinite training data for this problem. The number of humans that work on encryption is tiny. The idea that because humans haven't figured out how to break some encryption schemes it can't be done is kind of absurd.
You can build and train a model in about 15 lines of pytorch. And you can build and break your own 8 bit xor cipher in about 10 lines of python.
Hacker news is full of software engineers. You are unlikely to find one that hasn't built a model using pytorch these days, and an xor cipher is a common university lab exercise.
For any NN to learn, the function it is approximating must structured enough to admit small set of parameters (i.e., not exponential), otherwise it will take exponentially many nodes to do anything. AES is not differentiable, as are all secure hash and encryption functions. This is a basic test that is used to attack everything.
So you'd get a NN that must be big enough to simply memorize all plaintext, key, output triplets, which is simply a lookup table. With around 2^768 nodes. Good luck.