If that's the case, would a nice solution be to model it as a recurrent neural network, e.g. pytorch or jax, with a single convolutional layer, with a single kernel, which you don't backprop to, and then try to minimise to loss with reference to the target image, by backpropagating to the input image?
It then becomes highly analogous to Google's deepdream, but instead of optimising the image for the output class, you optimise it to the output image.