Author here - thanks for the thoughtful points! I think we agree on a lot, but do disagree on the conclusion :)

My mental model (mostly == Karpathy's "software 2.0" one) is:

    source code     :: training data   (and scoring function, if it's not simple MSE/MAE)
    compiler        :: training loop   (and network arch :: compiler flags, maybe?)
    compiled binary :: trained network (like the onnx file used in this demo)

So - trying to reverse-engineer a trained model would be analogous to trying to reverse-engineer a stripped binary, and reading the training dataset analogous to reading the source code.

The interpretability advantage I'm claiming only applies to complicated programs: for sufficiently (and equivalently)-complicated programs, changing input->output behaviors specified explicitly in a training dataset feels much easier than trying to change the implicit input->output behavior of the corresponding zillion lines of source code.

--- Specific points:

> Most datasets have a reasonable amount of wrong input->output pairs

Definitely true! My counterpoint would be: most code contains a reasonable amount of bugs.

> it’s pretty easy for it to cheat

Hmm, I don't find that CycleGan article worrying: researchers trained a network to pass information through a constrained channel, and it succeeded (just not in the way they wanted). Quoting the article itself:

> this occurrence, far from illustrating some kind of malign intelligence inherent to AI, simply reveals a problem with computers that has existed since they were invented: they do exactly what you tell them to do.

> your dataset is likely to be much, much larger than the equivalent program.

I agree! But (again, for complicated programs) I am comfortable sweeping over a large dataset for errors (or asking a team of people to do so, or writing code to find errors, or even training tiny networks to find errors). I am not confident I could debug the equivalent zillion lines of source code just by reading it. The effect on input->output behavior from a single dataset example feels obvious, whereas the effect on input->output behavior from a single line of source code may depend on me understanding the other zillion lines...

> Right now we haven’t developed much tooling to help you map a sample bad input or behavior back to training data that might be relevant

Agreed (sadly)... though I think root-causing misbehaviors of complicated codebases is also painful.

> I don’t think it’s currently true that ML systems are simpler or more interpretable than a corresponding imperative program.

I agree for simple programs, but disagree for complicated ones; for sufficiently-complicated programs, I would much rather deal with the dataset representation.