Ask HN: Does Google Index “Secret” URLs Only Shared Through Gmail? | Hacker News Reader

Ask HN: Does Google Index “Secret” URLs Only Shared Through Gmail?

3 pointsmacinjosh10 comments

I am seeing files I upload to S3 in Google search results. I am not sure how Google found the URLs to these files as I've only shared them in email (Gmail).

View on HN

Do you mind if I quote a couple of your articles as long as I provide credit and sources back to your website? http://www.juegosfriv2019.org/ My blog is in the very same area of interest as yours and my visitors would certainly benefit from a lot of the information you present here. http://www.juegosfriv2017.link/ Please let me know if this okay with you. Thank you!

Go into Incognito and see if they are showing up there. Google might show you files from Google Drive and other places because you are logged into your Google Account, but it may not show up for anyone else.

They do show up in incognito.

Well then, Google is just being Google and indexing everything. You might have to delete it or check your account settings to see if you can make it private.

Google definitely reads your email (gmail). As an example, if I have the word attachment written in my email and then attempt to send the email without any attachments, google will tell me that I used the word attachment in my email but didn't attach anything.

"Reads your email" is ambiguous. All unencrypted email must be read just to transmit it over the network, or to store it on disk. Sounds silly, but the example you gave is actually even more benign: It could be done with Javascript that never sends anything across the network.

You should put a robots.txt file in the S3 bucket if you don't want things in it indexed.

I (kind-of) doubt that they found the files through your e-mail, but the files in your S3 bucket are technically available for indexing since you don't have a robots.txt.

I do have a robots.txt, but for whatever reason its not found by Google

Are you using a custom domain? Robots.txt must be in your root folder, doesn't work with S3 urls.

yeah its the S3 URLs that are causing the problem

Reply on news.ycombinator.com