The big, important crawlers have known IPs don't they? Isn't that how news sites, quora and friends selectively apply their pay wall and still get indexed?