Graph the response of the neural network, over the range of the stimuli that you care about. This is going to be a ridiculously huge dataset, but bear with me. Then, use a genetic algorithm to evolve equations that have reasonably similar behavior, perhaps over a much smaller domain.
This collection of equations and their valid input ranges, are the raw material for your program. You would simplify them using algebraic solvers, when possible, and attempt to hand-optimize them for readability. Then, when you are done, the whole thing gets compiled down to big switch-case statement in, say, C. From here on, the process looks sort of like yacc.
So, by adding a whole new layer of magic, we get the system to explain itself in a way that a programmer could understand. Come to think of it, this feels sort of like how I do personal introspection. In fact, I'm doing it right now.