Definitely tune your robots.txt
But if you put exact URLs in your robots.txt, then a normal person could pick them up from there and access your content.
Make sure you use a directory structure and forbid all files from the content directory :-)
But if you put exact URLs in your robots.txt, then a normal person could pick them up from there and access your content.
Make sure you use a directory structure and forbid all files from the content directory :-)
Some other crawler may walk in there, then create pages that link your files, and then Google could link those pages, and it's game over.