"What does Googlebot see when it accesses my page?" | Hacker News Reader

"What does Googlebot see when it accesses my page?"

googlelabs.com

27 pointsjimmybot10 comments

This may not be all that useful for hackers who aren't doing anything particularly weird. Wget should return the same thing. The times that this is interesting is when wget doesn't return the same thing. This could indicate a bug in your ip targetting, A-B testing, or browser-specific delivery. It might also help reveal problems like mentioned here http://news.stepforth.com/blog/2006/09/has-your-site-been-ha... where a hacker shows regular users the original page and shows googlebot a page filled with links to porn.

Really not too helpful, pretty much a "view source" of your page as far as I can see.

If you have a simple application, then, yes, it's not going to give you a whole lot of new information. However, even in the basic case, you do get the HTTP response headers. That can be helpful sometimes in debugging problems.

The other reason why this is useful is that in larger complex deployments, there may be requests that get throttled, dropped on the floor, etc. as an automatic means of shedding load. We have such a mechanism at StyleFeeder, because we have so much robot traffic. Other times, load balancers and the like will inject stuff into the HTTP request stream in a way that isn't obvious when you're working in a dev environment.

I tried it, and got the exact same I'd get running wget against my site?

Are they advocating specializing pages for the Google bot?

Yeah, what exactly does this provide above "View Source"?

Almost certainly not - that's generally not a good idea and can result in a penalty if the content differs from what a normal human would see.

Hmm, for my site it seems to cut off at right around the 3000th line of source. Should I be worried about this? Is this a bug in the "Fetch as Google" feature or perhaps googlebot cuts off after the 3000th line...?

Awesome feature. I just wish you didn't have to have a GWT account to use it. A lot of webmasters who would have run it out of curiosity (and possibly found something wrong with their website) aren't going to bother.

If you don't have a GWT account, you're missing out on a lot of information that you would either (a) not ever see or (b) have to generate yourself.

I take it you can only look at your own sites?

Reply on news.ycombinator.com