Hi there,

Thanks, thats a good point! I will modify the crawler to respect those items, and also listed them as nofollow (similar to blocked by robots).