Show HN: GitHub News, using a peak detection algorithm to detect trending repos | Hacker News Reader

Show HN: GitHub News, using a peak detection algorithm to detect trending repos

gitlogs.com

184 pointszthomas31 comments

Developer here, I subscribe to Github's official weekly newsletter and various other ones but always found that they did a poor job at highlighting what's trending. The "top trending" repos always seem to be most established projects like Bootstrap or ReactJs.

I'm using a peak detection algorithm to find which repos are spiking in traffic and rank the uptrend based on a normalized growth curve. This filters out all the well established repos and actually uncovers new trends and launches. Hope you guys find it useful!

Some notable features, there's also a better interface to search topics on github as well as a curated topics section: http://www.gitlogs.com/resource-guides http://www.gitlogs.com/search

Appreciate any feedback,

This is awesome. I had the same desire and even told folks at GitHub that improving this report is low hanging fruit. It's been on my list of ideas to hack on for awhile :-)

Ideally you (well, really I :-P) also want star/follow activity from people you follow bundled in any update that's also reporting the projects discovered by your peak detection algorithm over that same period of time. There'd likely be some overlap in the report but you could indicate that in your report. This also gives you more incentive to login with GitHub which is a little lacking right now.

Anyway, thanks for doing this and great job :-)

Thanks for the feedback, the first thing on my list is definitely more personalization. I just wanted to get it out there and see if people liked it first.

This is cool! Any particular reason why you need permission to write all my public repositories when I log in?

Thanks, it's for starring a repo. Github ties it to that same permission scope. You can go to your Github integration settings and disable that scope just for a certain app if you want.

Awesome product :) Is it possible to allow accounts so that user can browse starred repos from your web app ? I am suggesting this because the search feature's topics looks quite complete already, so it would be great to navigate hundreds of starred repos based on the topics, which is something I've been looking for.

You mean to browse repos that you starred yourself? Yup probably pretty easy to do.

Rate Limit Reached:

You have reached the Github anonymous user limit, please login with your Github account to continue.

Oof. I was just scrolling down the front page.

---

Edit: After I signed in, I got a bit lower, then got the same message again (telling me to sign in)!

Make / break suggestion: File the stats on a local server. Don't expect the user to query Github or you're going to drop fans fast.

Hi DC2, actually I do cache some of the results, first 10 days and top 50 repos per day. I think I didn't expect people to scroll as far as they did. I'll up that cache later

Thanks! I was trying to find my repo, which trended about 15 days ago :)

Otherwise, it looks really good, and the algorithm results are refreshing.

Good name. Personally I would like to receive a weekly digest for one or two specific languages.

Thanks man, yup. Definitely on my todo list.

I can recommend you to "split" your stream of repos in multiple streams [1], and then create a function to merge multiple streams into a single one.

This way you can gain: 1. detect spikes in less popular streams, and decide via the merging function how to compare them to spikes in popular streams. 2. personalization, where each user can potentially get its own stream combining the categories she's interested in.

[1] e.g., by language as also suggested elsewhere, by "category" if you can come up with some definition, potentially by source (e.g. big company) as well.

If this allowed you to filter by language it would be infinitely more interesting.

You can do that in the search function right now, in the future I'll add it to the news section as well. http://www.gitlogs.com/most_popular?from=2016-03-03&language...

Can one fork this on GitHub?

I'm interested in the peak detection algorithm.

Why are you asking for "read and write all public repository data" & "read and write all user data."? I understand the read, but not clear on the write

I think someone else mentioned this as well, but unfortunately Github ties the "staring a repo" to the write permission. I think I can try to separate it out so it can ask for that scope when a user wants to use that feature.

Thanks for the response! Github is definitely a little unclear on the permissions side of things. Might help clear things up if you mention what you are doing with the permissions somewhere.

You get the "Success -- Thanks for subscribing" message even if you do not put an email id in the input field! Please correct!

Pff. That's a vicious circle. Then trending repos will get even trendier. Social networks are largely enough to expose that kind of "noise"/"buzz". I think that a good metric to find interesting projects that are starting or are not already under the spotlight would be to use the ratio: new visitors / stargazers. Quite hard to achieve but I really think it would be more accurate (and fair).

That's definitely something on my mind originally. But since I can't get traffic data from Github, the best thing I can do is to filter out the highly established repos and weigh new repos higher in the rankings. You can see that even with just 150+ stars gained for a brand new repo this system will show it at the top top of the rank despite other popular repos that might have gotten 700+ gains on that day.

However, that said I do plan to improve discoverability even more with traffic data since now I can atleast measure views on gitlogs.

I'd love to have this as an RSS feed.

Is the source available? I'd like to do this on a small subset of repos I watch, about 2k in total.

Cool concept, would love to see graphs over weekly/monthly timespans or something like that.

Thanks, I actually did build something like that. My original inspiration was Google Trends (thus the UI) but the data wasn't very useful, traffic on github is very spiky, even on a log graph it didn't look good.

I don't think you should be title-casing the repository names.

Why does not this works with localStorage disabled?

Hey thanks for the tip, it took me a while to find it but apparently react-router 1.x had an issue https://github.com/reactjs/react-router/issues/1661 and I had to upgrade to react-router 2.x I'll push out a fix soon

gitlogs

Where git = github

Looks great though! Already found something sweet.

Another feature request, can you add an up/down trend indicator?

Can I ask which peak detection algorithm you used?

Right now if a repo is trending down (from the moving average) it will have a negative score and won't be showed at all.

The algorithm basically looks at the moving average, the new watchers gained and the number of total watchers. It'll calculate the the rate of change in new watchers and compare that against what it expects given the lifecycle of where the repo is at. So a brand new repo adding 150 stars per day will beat out an established repo that adds 250 stars per day.

Reply on news.ycombinator.com