The code and weights can be made available. He can even single step through the code. His ability to understand it though might be limited by his own intelligence.
You can't be serious. As if going line by line would make it any clearer why the software is doing what it's doing when it's obfuscated with so many neurons and layers that there's no way to grasp the entirety of the system through code alone.
My point is that it’s not really a black box. You need to understand it like you need to understand the brain. Our inability to understand the brain says less about the brain and more about our own intelligence (which is housed in the brain, so it gets a little circular here).