When I had this happen, or when I heard this happen, there was always a reason that was far less sinister than initially thought.
* Public referrer logs created a backlink
* Somebody (else) published the URL
* Somebody (else) shared the URL
* You pinged the URL to search engines or other services.
* The URL appeared in your RSS feed
* The URL appeared in your sitemap
* Your pages URL ranges are easily guessable (/item.php?id=1007, /item.php?id=1008) and traversed by a search engine.
And more recently, something less innocuous: You simply added a Google +1 button to your pages.
When you add the +1 button to a page, Google assumes that
you want that page to be publicly available and visible
in Google Search results. As a result, we may fetch and
show that page even if it is disallowed in robots.txt.