Why not use the GPU? This is exactly the kind of tasks GPUs were designed for. Gimp can do this in real time using the difference layer mode.
There's also bound to be some vector compare trick as well which falls back to pixel-by-pixel only for differences.
I think there are quite a few simple tricks you could apply here to get a bit more performance... Starting with sacrificing some memory and decoding a chunk of image at a time rather than calling .At(x,y) every time. And maybe improving local cache by dividing the image in `cpu` parts rather than having the work interleaved.
I would guess that the time of getting the data out to the GPU and back, is about the same as just calculating the difference on the CPU.
So, the quickest solution is probably to use the CPUs vector operations, and read ahead into the registers so you can exhaust the memory bandwidth fully.
The actual file transfer is comparable to the GPU doing a memcpy. (You could also use pinned host memory if the GPU program is just doing a one-off read.)
GPU processing for simple operations really shines if your data is already on the GPU and your code is hot. Otherwise, it's more useful if the process is significantly complex enough to make the overhead of a cold startup insignificant.