Curl and Cloudflare = Information leak – But why?

> Update #3

> If you care about your unlisted URL privacy with Cloudflare, make sure you don't have the Crawler Hints feature enabled. - A good hunt, thanks and sorry!

How about not using CloudFlare at all?

Quoting from CF's own blog [0]:

> Going forward, because the benefits are so substantial for site owners, search operators, and the environment, we plan to start defaulting Crawler Hints on for all our customers.

Which by the way directly contradicts what jgrahamc stated in this thread.

[0] https://blog.cloudflare.com/crawler-hints-supports-microsoft...

Not a leak, this is how IndexNow works, if you don't want this, disable Crawler Hints.

It could also be a site map[1] which is auto-generated by a static webpage generator (Hugo does this IIRC)

[1]: https://developers.google.com/search/docs/crawling-indexing/...

Apparently bing webmaster tools will tell you who submitted the URL if its through IndexNow. Might be worth a shot.

If you cant find it this url may help (did not test): https://www.jezweb.com.au/incorrect-urls-in-bing-indexnow-fr...

Most browsers have safe browsing features that send unknown URLs to a cloud service for verification. I was shocked once to see an S3 presigned URL be accessed from Russia and India, when I had not shared it yet. It turned out to be Google safe browsing service (this case was several years ago and was verified with Google; I don’t know where URLs are investigated these days).

Another possible culprit is browser extensions. If you give an extension permission to a URL or page content, you have no idea if or where it’s sending it. Some extensions intentionally send URLs to cloud services for benign purposes (ex: Microsoft safe browsing extension for non-MS browsers).

I imagine this is Crawler Hints (https://blog.cloudflare.com/tag/crawler-hints/). If OP has Crawler Hints enabled on their domain then Cloudflare will let search engines know about pages.