Reminds me of the term 'system identification' from old school control systems theory, which meant poking around a system and measuring how it behaves, - like sending an input impulse and measuring its response, does it have memory, etc.
How do you handle that sort of thing? Maybe main process then leave some relatively small residual to the NN?
Is your poking more like "fuzzing", where you just perturb all the input parameters in a relatively "complete" way to try to find if anything goes wild?
I'm very interested in the details behind "critical" type use cases of NN, which I've never been able to stomach in my work.
Faster time to market/production is for us the main reason/advantage of the approach.