Handwriting a GST pipeline is pretty much what I did. I start with frame differences(I only decode the keyframes that happen every few seconds, so motion detection has to work in a single frame to have good response time).
Then I do a greyscale erosion to suppress small bits of noise and prioritize connected regions.
After that I take the average value of all pixels, and I subtract it, to suppress the noise floor, and also possibly some global uniform illumination changes.
Then I square every pixel, to further suppress large low intensity background noise stuff, and take the average of those squares.
I mostly only run object detection after motion is detected, and I have a RAM buffer to capture a few seconds before an event occurs.
NVR device code(In theory this can be imported and run from a few like python script), but it needs some cleanup and I've never tried it outside the web server.
https://github.com/EternityForest/iot_devices.nvr/blob/main/...
GST wrapper utilities it uses, motion detection algorithms at top:
https://github.com/EternityForest/scullery/blob/Master/scull...
My CPU object detection is OK, but the public, fast, easy to run models and my limited understanding of them is the weak point. I wound up doing a bunch of sanity check post filters and I'm sure it could be done much better with better models and better pre/post filtering.