And, to at least respond to two obvious counter-arguments I've looked into:
"But it's just one line of code to change it to scan images (that aren't uploaded to iCloud) (that are anywhere on the device)!" No, it isn't; if you read the technical documentation and the more technically-oriented interviews Apple's given on this, there isn't just one hash that needs to be matches, there are two hashes, one on the device and one on the server. (I think Apple did a very poor job of communicating this to a general audience; it certainly wouldn't have alleviated all the concerns, but if it was understand as "client-server scanning" rather than "client-only scanning" it might have at least changed the tenor of the conversation.) That doesn't mean they can't do a combination client-server scan on every single image or even file on the device, but it makes it both more difficult to do and more difficult to hide.
"But what if the system doesn't work as Apple's described it?" Well, if you don't trust Apple to some degree, all bets are off. They already do ML-based image analysis of all photos in your photo library regardless of iCloud status and they've literally demoed this on stage during iPhone keynotes, so if Apple was going to secretly give government access to on-device scanning, a different technology -- one that works (questionably well) on all images, not just already-learned ones -- is literally already there. The only way you "know" what Apple is doing with your data on or off device comes from a combination of what they tell you and what third-party security researchers discover.