Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by bmurray7jhu | Hacker News Reader

Full thread

bmurray7jhu·

View on HN

>Disallow: /polska

Why is only poland blocked for all robots? Why is there no /deutchland?

What's this: https://www.bloomberg.com/polska

Seems that Bloomberg hosts something for the Polish, but I'm not sure what.

Financial reports about companies, not for their average readers

Looks like aggregation of press releases coming from public companies listed on Warsaw's stock exchange.

That I, Robot (Asimov) reference though. [1]

[1] https://en.wikipedia.org/wiki/I,_Robot

> If you can read this then you should apply here.

If you also know why we're making some basic mistakes here, especially apply here.

I don't know much about the content of robots.txt, can you share some mistakes they are making here?

It's not really a good idea to restrict crawlers at directory level since it doesn't prevent any of those pages being indexed. You can get weird behaviour such as googlebot trying to crawl a page but not being allowed to. If this happens a lot then they are going to penalise you for it because they don't want a bunch of pages in their index that they don't know the content of. If this number of pages is a significant percentage of your site then you are in real trouble.

Much better to use x-robots noindex, nofollow for pages you don't want to be in the public domain.

https://developers.google.com/search/docs/advanced/crawling/...

... if you're specifically targeting Google.

If you are a neutral site (in this case, caters to a lot of people - especially in Asia, where Google as a search engine is a hit-or-miss proposition), then this might backfire sine other crawlers won't understand X-Robots. You can mitigate this by knowing which spiders knows X-Robots (Google, Bing and Yandex AFAIK) and whitelist them while still using a disallow directive for the rest of them.

Reply on news.ycombinator.com