Comment by chimeracoder | Hacker News Reader

Comment by chimeracoder | Hacker News Reader

Parent Full thread

This is an interesting idea, but it suffers from one maor problem: Github's language detection is rather inflexible, and sometimes misleading.

For example: I have a number of small Go projects that are incorrectly categorized as CSS or Javascript. One such example is a project explicitly designed to be a Go webserver that you can just "go get" (or git clone) and run immediately, with no extra setup, sort of like Twitter Bootstrap, but for the backend: https://github.com/ChimeraCoder/go-server-bootstrap

Because it self-hosts the Bootstrap files, and because the lines of Go code required are relatively small, less than 5% of this project is detected as being written in Go, even though I didn't write a single line of Javascript or CSS myself for the entire project.

(Yes, I know I could pull in the Bootstrap, etc. from a CDN, but this is just to illustrate the limitations of an automatic language detection model that the developer has zero control over).

> an automatic language detection model that the developer has zero control over

Pull requests welcome :)

https://github.com/github/linguist

I've seen that, and this is the corresponding issue: https://help.github.com/articles/my-repository-is-marked-as-....

Looking at vendor.yml, I'm actually not sure why this is happening. The repository I linked to above only has two files that end in .js: bootstrap.js and bootstrap.min.js[0], and only one that ends in .css (bootstrap.css). Unless I'm misreading the regex in vendor.yml, it seems like they should be excluded[1] but they aren't, and this isn't the only project of mine for which this is happening.

I'm fairly certain it's picking up those files and not miscategorizing the Go files for some reason, because when I delete those files and instead serve the CSS/Javascript from a public cache, it categorizes the repository correctly, as is the case with this project (based off of go-server-bootstrap): https://github.com/ChimeraCoder/pluto

[0] https://github.com/ChimeraCoder/go-server-bootstrap/tree/mas...

[1]https://github.com/github/linguist/blob/master/lib/linguist/...

So the jQuery regex, for comparison, is

  (^|/)jquery([^.]*)(\.min)?\.js$

while the Bootstrap regex in question is

  (^|/)bootstrap([^.]*)(\.min)\.(js|css)$

with the ".min" not optional. (Looks like this was the intent [1]).

The 68.9%/26.5% percentages match the 130k/50k ratio of bootstrap.css to bootstrap.js, so it looks like only bootstrap.min.js is being excluded?

[1] https://github.com/github/linguist/commit/d5002ef06a9f346391...

Thanks for pointing that out. You're right - it does seem like it was intentional. I filed an issue for this though just in case, and perhaps they'll explain the reason: https://github.com/github/linguist/pull/856

Linguist also has an odd tendency to miscategorize projects as being written in Prolog, thus inflating the repository rates of an already very niche language.

Perhaps it has something to do with (at least in part) Prolog and Perl sharing the same extensions?

Same problem happens with Rebol which often get categorised as R on Github (both share .r extension).

some ref: https://github.com/search?l=R&q=rebol | https://github.com/search?l=R&q=rebol&type=Code

Reply on news.ycombinator.com