TensorFlow Lite with SSDLite-MobileNet gets you around 4 fps on a Raspberry Pi 4 (23 fps with a Coral USB Accelerator): https://github.com/EdjeElectronics/TensorFlow-Lite-Object-De...
The better approach is to use threading to keep all the Pi's cores busy and the USB accelerator busy at the same time, and to use hardware acceleration.