But gitweb is probably the second most used method of hosting a git repo and easily recognizable through heuristics. If it’s gitweb, fallback to git access and save everyone, including the crawler, time and resources.
Understood. The comment I replied to suggests the crawlers should figure this out on their own.
No, when I said this behaviour is stupid I did not imply that it needs to be solved with "intelligence". This class if problem is already solved by traditional crawlers, AI companies just actively chose to disregard any of that in their race to the top / bottom.