You will need to disallow all and enable specific search engine crawler ids

Something akin to https://stackoverflow.com/questions/59092667/allow-only-goog...