Soon you won't find the Language Benchmarks Game with any search engine | Hacker News Reader

Soon you won't find the Language Benchmarks Game with any search engine

benchmarksgame.alioth.debian.org

48 pointsgus_massa15 comments

Open articleView on HN

If only there were other places to locate web sites on the Internet...

why don't you put it somewhere else?

That, or give the benchmark game a separate config to allow googlebot greater/faster access.

I would change hosting providers.

Block everything in robots.txt except the homepage, instead ?

The solution they chose seems to be the absolute silliest way to solve the problem.

I hope this throttling includes the VCS web frontends. It used to be that the hg (mercurial) and some other VCS (I forgot which one it was) web frontend were crawled mercilessly by someone (unidentified so far AFAIK). That used to put so much load on alioth that the git frontend would refuse to work, which was very annoying for me :).

That being said, I hope that either

a) the author of the post is exaggerating or

b) the alioth admins will improve the throttling to better target abusive users and not legitimate websites such as this one

a) I was not exaggerating

b) All the upstream Fusion Forge generated pages, for all 1,000 projects, were being served without any HTML robots meta tags -- which, of course, is asking for

and duplicate pages were being served from 4 subdomains, and the 1,000 projects also had their own web pages without any HTML robots meta tags...

In other words, googlebot was visiting alioth because alioth was repeatedly asking googlebot to visit.

Luckily I found a way for the alioth admins to take some control of the Fusion Forge generated pages, that did not require changes to Fusion Forge.

Header set X-Robots-Tag "noindex, nofollow"

https://developers.google.com/webmasters/control-crawl-index...

The alioth admins were willing to ease throttling and see if asking robots not to index and not to crawl Fusion Forge pages reduced the server load.

Currently server errors for benchmarksgame subdomain are below 70% and robots.txt below 5% and the benchmarks game pages are indexed.

Perhaps they could invent a benchmarking tool that would allow them to write more efficient code that would utilize fewer resources on the server.

"There's been no suggestion that the benchmarks game website was the primary cause of that server load. It's just the result of the default configuration they use to provide dynamic content for 1,000 projects."

If you wrote it, you'd have to get 999 other projects to bother to implement its results.

What I would like to see rewritten is a (obvious) JVM view. So that's all very interesting that a n-body simulator in Clojure takes 60 times the memory and 4 times the exec time of a C version or about half a meg of ram per body in the n-body problem (uh, what?) What would really interest me is at a Debian OS level what is the delta when I move an identical benchmark from a Debian box running -i386 to -amd64 or at least for the interpreted JVM family of languages what happens to speed and memory when I move from vers 6 to vers 7 or how about alternative JVMs or whatever. More of an OS tweaking focus than a language focus.

There is a startup lesson in this situation that anything shared can be a focus of pain, also any single point of failure can be a focus of pain. Not exactly new observations, but, still true. If instead of a benchmark site, this project was noobs trying to leverage the synergy of tweeting crowdsourced dog food delivery over the internet, they'd be dead meat once they disappear from search engines.

>>What would really interest me ... vers 6 to vers 7<<

All the measurements are in CVS so you actually could get at measurements for different versions.

>>What would really interest me ... alternative JVMs<<

Please take the program source code and the measurement scripts and publish your own measurements --

http://benchmarksgame.alioth.debian.org/play.php#languagex

>>what is the delta when I move an identical benchmark from a Debian box running -i386 to -amd64<<

Just open 2 web browsers and look ;-)

http://benchmarksgame.alioth.debian.org/u64q/measurements.ph...

http://benchmarksgame.alioth.debian.org/u32q/measurements.ph...

That would not stop bots crawling the pages generated by the hosting software, about 90 pages for each of 1,000 projects, duplicated at 4 different sub-domains.

This stuff --

http://alioth.debian.org/projects/benchmarksgame/

The X-Robots-Tag HTTP header does provide a way for admins to instruct bots not to index and not to crawl -- without requiring changes to upstream software.

So, move to a new host?

It's still the top result for "language shootout", despite the name being changed year(s) ago.

The website was renamed back on 20th April 2007.

http://benchmarksgame.alioth.debian.org/dont-jump-to-conclus...

Changing the website banners and page titles and ... proved to be too subtle, so a new project was created just so the website could have a benchmarksgame sub-domain.

The URLs were changed December 2012 and the switchover took a few days --

    16th 0 visits vs 1,048
    17th 265 visits vs 1,026
    18th 1,262 visits vs 136
    19th 1,333 visits vs 8

The old sub-domain scripts were changed to redirect --

    header("HTTP/1.1 301 Moved Permanently");
    header("Location:http://benchmarksgame.alioth.debian.org/");

Reply on news.ycombinator.com