You know, at this point, I wonder if an allowlist would work better.
1. A proxy that looks at HTTP Headers and TLS cipher choices
2. An allowlist that records which browsers send which headers and selects which ciphers
3. A dynamic loading of the allowlist into the proxy at some given interval
New browser versions or updates to OSs would need the allowlist updating, but I'm not sure it's that inconvenient and could be done via GitHub so people could submit new combinations.
I'd rather just say "I trust real browsers" and dump the rest.
Also I noticed a far simpler block, just block almost every request whose UA claims to be "compatible".
To truly say “I trust real browsers” requires a signal of integrity of the user and browser such as cryptographic device attestation of the browser. .. which has to be centrally verified. Which is also not great.
Forcing Facebook & Co to play the adversary role still seems like an improvement over the current situation. They're clearly operating illegitimately if they start spoofing real user agents to get around bot blocking capabilities.
That could also be a user login, maybe, with per-user rate limits. I expect that bot runners could find a way to break that, but at least it's extra engineering effort on their part, and they may not bother until enough sites force the issue.