Ask HN: Are search engines indexing short-urls? | Hacker News Reader

Ask HN: Are search engines indexing short-urls?

1 pointSeldaek4 comments

Now I do wonder. All those tweet containing links to a site. They all use bit.ly and whatnot. Fine. But what does Google (or others) do when it sees such a link? From what I can see in Google Webmaster Tools, it seems that all these links don't count as links pointing to the site. A search for "link:someurl" also doesn't seem to return any result from twitter. This means you get no pagerank from twitter traffic.

Any idea how to fix this? Specifying rev="canonical" maybe?

View on HN

You could find out if the short URL service is using a robots.txt file to block search engine crawlers.

Next you can look what kind of redirect the service is using: status code 301 or 302 or something unusual like 307. Rex Swain's HTTP Viewer is a nice tool to find out the status codes: http://www.rexswain.com/httpview.html

A site: search on Google for the domain of the service may also be interesting.

I have seen that Googlebot and other crawlers will also follow "nofollow"ed links, I suspect that they use these links for page discovery like ordinary links. The nofollow attribute just tells the search engines that this link can't be trusted and shouldn't be considered for ranking calculations (like PageRank etc.).

It is not clear how much value Google and other search engines give to a link that goes through a redirect. Sometimes pages redirecting with the code 302 appear in the search results instead of the target page (302-hijacking).

The link: operator is broken, it shows you only a small selection of links pointing to a site if anything at all. The Google webmaster tools show more links, but not all of them. Yahoo Site Explorer shows more links: http://siteexplorer.search.yahoo.com/ I have seen nofollow links in both of these tools. They do not tell anything about the value of a link.

I remember that I have seen links from tinyurl in the old version of the Google Webmaster Tools.

Twitter puts a nofollow attribute on its links to reduce incentive for spammers to tweet. Nofollow means search indexers don't crawl from that link nor count it toward any ranking. (Twitter would have low SEO values anyway because it has too high a concentration of links compared to text.)

As for short URLs, they shouldn't be designed to steal page rank. If they do a proper 301 redirect, then the original address should receive the SEO value.

Yes, bit.ly which is probably the most used service on twitter does proper 301s as far as I know, so this isn't the issue. Good point about nofollow though, I completely overlooked that. Cheers.

Yes they index these links. The "problem" (if you can call it that) is whether the crawlers discover the links or not, and if they do, how much of a priority they are.

For Twitter specifically, remember that lots of websites syndicate tweets, links and all. It's likely a crawler would discover the links on non-twitter.com websites.

Reply on news.ycombinator.com