Show HN: HN Trends - like Google Trends, but for Hacker News | Hacker News Reader

Show HN: HN Trends - like Google Trends, but for Hacker News

hntrends.jerodsanto.net

355 pointssant0sk1106 comments

Hey Jerod. I own the domain HNTrends.com, which I used for a similar project about two years back [1][2]. If you plan on keeping this project up, I'd be happy to transfer the domain name over to you. Shoot me an email: matt@leandesigns.com.

[1] http://news.ycombinator.com/item?id=810112

[2] http://www.mattmazur.com/category/hntrends/

These trends are fairly interesting...

http://hntrends.jerodsanto.net/?q=pg

http://hntrends.jerodsanto.net/?q=y+combinator

http://hntrends.jerodsanto.net/?q=pg%2C+tptacek%2C+patio11%2...

http://hntrends.jerodsanto.net/?q=2007%2C+2008%2C+2009%2C+20...

http://hntrends.jerodsanto.net/?q=android%2C+iphone vs. http://www.google.com/trends?q=android%2C+iphone&ctab=0&...

http://hntrends.jerodsanto.net/?q=show+hn%2C+ask+hn%2C+tell+...

http://hntrends.jerodsanto.net/?q=mac%2C+gruber%2C+steve+job...

Don't forget thought that these trends include the user's own comments and submissions in addition to mentions of them in other users' comments.

suking, this is truly awesome. It immediately joined my bookmark folder of favorite HN tools. I'd be curious, though, if whether or not it would be possible to filter out user names when counting how many times a word has appeared on the site. Or would that be too much of a challenge?

I was surprised to learn that the search API returns hits on usernames when they appear outside of the post/comment content.

I'm looking now to see if there's an easy way to filter around it. I've also been emailing with one of the ThriftDB guys and I'll ask him as well.

If not, it'd be very difficult to filter those out because I'm not currently (nor would I want to) retrieving all the matching items, just the number of hits.

Oh yes, I forgot you were using the HNSearch API. By the way, I hope you win the API Contest (don't they decide tomorrow?). You for sure get my vote!

Awesome, thanks! The voting starts tomorrow. Do come back and participate ;)

This is actually pretty easy to do! You can apply weights to fields, so just apply 0.0 to the username field - or that's how I did it anyway.

Your entry looks very nice, cool to see others had a similar idea. :)

What else is in that bookmark folder of your favorite HN tools?

Here are some of the various HN tools and interfaces I have collected over time:

hckr news - http://hckrnews.com/

Sortable HN - http://hnsort.com/

hnvue - http://www.hnvue.com/

HackemUp - http://www.mrspeaker.net/2011/04/11/hackernews-bookmarklet/

Search Y Combinator - http://searchyc.com/

WWHNS (What Would Hacker News Say) - https://github.com/marcua/wwhns/

HNSearch - http://www.hnsearch.com/

Visual Search - http://hnsearch.heroku.com/#h

Hacker News Resources - http://hnresources.com/

Kalimba - http://hn.embed.ly/

The Best of ... Hacker News - http://bitovod.com/hn/best-of

HackRSS - http://www.hackrss.com/

HN Trends - http://hntrends.jerodsanto.net/ ;)

Hacker Follow - http://hackerfollow.com/

HNInstant - http://hninstant.com is a better hnsearch

About to hit issue #52 and 3000 subscribers with my weekly Hacker Newsletter project - http://www.hackernewsletter.com

Yes, this is another HN resource that I forgot to mention. In fact, I just discovered last week and decided to sign up. I really enjoyed it - keep up the good work!

Thanks and glad to hear that!

http://hntrades.com/

Sounds like PG/YC should launch a hacker news app store now. :)

It seems that this one has just arrived: http://hnmood.com/

Hacker news daily - http://www.daemonology.net/hn-daily/

Some rather social search terms:

http://hntrends.jerodsanto.net/?q=they%2Cwe%2Cyou%2Ci

I don't understand why "they" doesn't occur. Maybe a bug?

http://hntrends.jerodsanto.net/?q=they+deserve%2Cwe+deserve%... http://hntrends.jerodsanto.net/?q=they+should%2Cwe+should%2C... http://hntrends.jerodsanto.net/?q=they+failed%2Cwe+failed%2C... http://hntrends.jerodsanto.net/?q=i+am%2Ci+was%2Ci+will+be

Idle speculation: Wouldn't it be cool if we plotted a user's total karma next to the trend of their name and it turned out to look like an integral?

For some of us, it will look more like a martingale :-).

Can not click the point to view all the relative articles, but, it's sort like painting and counting looks so cute, ^_^

first of all, very cool.

how is it normalized? every single thing I type in has an upward trend, which to me just suggests that interest in Hacker News has increased over the last few years... and I don't really need a chart to tell me that since I live in silicon valley :)

Good point. I wasn't really thinking about normalization as I was more interested in comparing trends on multiple terms, but I don't think it'd be too much work to change the algorithm to account for HN's growth.

Yah, you could easily do it by total # of posts to account for the growth. Cool stuff man.

The trends on multiple terms are actually easier to read after normalization anyway. Right now the power-law curve growth of HN in total is making it hard to read the difference in growth between, say, FB and twitter. Easier for our brains to process if you took the ln() of them - which is definitely the fastest way to fix the charts - or normalized in some other way.

Came to say exactly this. The chart seems to start (by default anyway) at the beginning of 2007. I'm guessing the user-base and thus activity of HN has increased considerably since then. So yes it would be good to be able to re-normalise in some way. By the number of registered users would probably be the simplest. By number of comments posted would be another, by number of submissions or even number of words would all be interesting.

Btw - my suggestions got me thinking - I assumed this is looking at posts and comments. Whether just submission titles or if includes comments will affect best way to normalise.

It uses submission titles AND comments. How would you suggest going about normalization?

In that case I would do it by number of submissions (posts) + number of comments for each period (3 months I think, looking at your graphs?). I would normalised all quarters to the last complete quarter.

So if say in q1 2011 there were 1m posts + comments, and in q1 2007 10k, multiply the count for q1 2007 by 100.

It would be truly great to have this normalisation optional, e.g option to normalise by comment+post count and maybe also normalise by number of users (could be an interesting comparison). And perhaps have them super-imposed on each other!

Thanks, I will definitely implement the multiplier as you suggested. I like this better than showing as a % of total.

I'm storing the total hits so I will do the calculations on the client-side, which means I just might be able to add some toggles for the normalization schemes ;)

Thinking a little bit more about this, there are of course lots of submissions that don't get voted up or commented on - spam, off-topic, not interesting etc. Maybe you should only use posts that have either at least one upvote or comment, just to filter out the crap.

A first version of this has been implemented. You can't toggle the normalizations just yet, but I figured I should get the fix out there as soon as I could.

Check the "Learn More" section where I give you props for the help :)

Awesome, and thanks :)

Also, see my other comment below for a suggestion re filtering out crappy posts.

Coincidentally, I was also working on an HN Trends app a couple weeks ago (though I got distracted and haven't finished), and mine is normalized (by number of comments). For example: http://hn-trends.heroku.com/trends?q=scala%2Cclojure

Your app looks a lot better, though! (And my dataset ends around Sept. of last year.)

Divide by the weighted average change in the number of hits for the 100 most common English words? Basically, add up all the uses of "and," "or," "a," "the," etc. in each period. If Q3 2009 has 4X the sum of those words as Q1 2007, then divide all Q3 2009 results by 4.

Unfortuantely, many common words seem to be filtered by the API. Some however are not, and they show a common growth pattern: http://hntrends.jerodsanto.net/?q=what%2C+you%2C+here

Doesn't seem to be normalized at all. The easiest way to fix would be to chart the y-axis on a log scale.

Just an update to let you know that the site now accounts for HN's growth. I used retube's suggested adjustment scheme. Let me know what you think!

http://hntrends.jerodsanto.net/?q=fuck%2C+shit%2C+cunt%2C+co...

I put in 5 of the 7 George Carlin dirty words. Looks like 'motherfucker' has a recent surge.

http://hntrends.jerodsanto.net/?q=piss%2C+tits%2C+cunt%2C+co...

If you want to see it without the top two words dwarfing the other five.

Possibly from "The Motherfucking Manifesto For Programming, Motherfuckers"?

http://hntrends.jerodsanto.net/?q=ruby%2Cpython

http://hntrends.jerodsanto.net/?q=rails%2Cdjango

Interesting that Ruby stays below Python but Rails stays well above Django.

There are similar metrics for job openings. It seems plausible that Python is becoming an all-purpose language while the focus for Rails / Ruby is on web apps. If true, this would be reminiscent of Perl v. PHP.

we may be in a bubble about talking about a bubble: http://hntrends.jerodsanto.net/?q=bubble

http://hntrends.jerodsanto.net/?q=bubble%2C+fuck interesting...

a very nice tool!

Hah, I think that's a neat way to visually normalize against the traffic increase trend.

http://hntrends.jerodsanto.net/?q=android%2C+iphone

http://hntrends.jerodsanto.net/?q=android%2C+iOS

http://hntrends.jerodsanto.net/?q=google%2C+apple

These trends don't account for traffic increasing to the site (and therefore posts/comments) as time progresses. That makes this less useful.

Agreed would love to see a normalization option that transforms it into a % of submissions in each period.

Nice work! I had the same exact idea for the contest and bought the hntrend.com domain 2 weeks ago, too bad I was too lazy to finish it :)

well, looks like there is a service which would deserver that domain...

http://twitter.com/#!/sant0sk1/status/82564569708503040

Had the same idea myself as well, bought hnstats.com (although I admit, I looked for hntrend and hntrends.com)...not enough time in the world to do all of these things. Great job with this.

On an IPad1 the charts render so slowly it's watching a race between brightly colored snakes. Very suspenseful.

Please do normalize the results.

The charts are SVG and iOS's implementation of SVG is stupid slow.

It works great on the iPad 2' with animations and everything

I searched for my own username expecting a flatline, but there were a few hits in the last few years. Flattering, but I'm fairly certain that hackers have not been talking about me. ;)

Perhaps you should only query titles and text, not usernames?

edit: for example, searching for pg would include his own submissions as well as mentions and Ask PG subs.

Strange, I didn't expect the search API to return results for a name that's in the by-line of a comment, but it sure does:

http://www.hnsearch.com/search#request/comments&q=pvilch...

This is something I should definitely work around in the future. Thanks for bring it to my attention!

You can exclude authors by setting the weight to 0 in your API call:

&weights[username]=0.0

Perfect, will do. Thanks!

I found this interesting: "Hacker News" vs. "HN"[1]

As the site gains reputation/users you can see that many switch to the abbreviated version.

[1] http://hntrends.jerodsanto.net/?q=hacker+news%2C+HN

http://hntrends.jerodsanto.net/?q=scala%2C+erlang%2C+clojure...

interesting indeed

It would seem that NodeJS is taking away some of the share of functional programming languages.

Let us never forget the first great hacker news bubble.

http://hntrends.jerodsanto.net/?q=nickb

Hi, I created a similar tool using the search API, but with lniks to the top stories (by points) over time

http://touchwoodsoftware.com/yctrends.html

Feel free to take that idea if you like, I think it's interesting not just to see the shape of the trend but also the stories which created the peaks.

inb4 http://hntrends.jerodsanto.net/?q=bitcoin

similar results.. http://hntrends.jerodsanto.net/?q=airbnb

Technews trends, what are we reading and sharing here: Techcrunch vs Mashable vs Readwriteweb vs CNN vs Techmeme

http://hntrends.jerodsanto.net/?q=techcrunch%2C+mashable%2C+...

An observation about geographies: http://goo.gl/4BUFL

It's cool but doesn't always seem to chart the trend. I have to force a refresh. Using Chrome stable.

I like the animation, but it will be an improvement if the chart max stays and not jump up and down

The problem with that is that I have no idea what range the terms queried will be in beforehand. Any ideas how to accomplish this?

You could animate after you've received all the data, then set the Y axis into the max of the values. Anyway, good job! Thanks for sharing.

Pretty cool visualisations. Just out of curiosity, why did you choose HighCharts? Did you look at jquery FLOT and choose HighCharts over FLOT? Or were you unaware of FLOT altogether?

http://code.google.com/p/flot

I chose HighCharts because I have experience with it and knew right away that it'd get me where I wanted to go.

I had never heard of FLOT. Thanks for introducing me to it, it looks cool and I'll check it out for future projects!

For the people who says that HN discuss less about startups, the trends actually shows something else: http://hntrends.jerodsanto.net/?q=startups

(Cross-post) There could be a spike in HN Trend's traffic http://twitter.com/#!/timoreilly/status/82824209721597952

What the hell happened with Javascript in mid 2009?

http://hntrends.jerodsanto.net/?q=coffeescript%2C+jquery%2C+...

Node.js was released in mid-2009. Did it launch a surge in JavaScript popularity or just ride it?

Is it possible to somehow normalize the numbers by the total number of posts/comments, so that trends across the years do not mostly reflect the user base/activity growth at HN?

Awesome work thus far though.

Yup, this seems like an obvious improvement as I believe you're the 3rd to mention it already :)

I'll definitely do that next, since it's not too hard to implement and seems like a big win when you are just trending a single term.

Very, very slick. And now I can can graph my increasing time spent on HN :)

http://hntrends.jerodsanto.net/?q=jamesbritt

I asked it about ios and android and it broke :( http://hntrends.jerodsanto.net/?q=ios%2C+android

Should be back now. I had to restart the heroku process. Sadly, no indication why it crashed :/

You may find this to be a more useful comparison: http://hntrends.jerodsanto.net/?q=iphone%2C+android

Out of curiosity: http://hntrends.jerodsanto.net/?q=ycombinator%2C+techstars

Obvious causation: http://hntrends.jerodsanto.net/?q=dolphins%2C+PROTECT+IP

Awesome work!

Can you normalize it against total text volume? Number of mentions always goes up with time as HN grows, % of posts mentioning will be a better measure of mindshare.

Thanks so much! Hmm, good idea on % of posts, though most of the value I believe is in comparing multiple terms, in which case the raw number of mentions isn't as important as the relation of the trend lines.

That's not entirely true.

For example, the first thing I searched for was "Haskell", curious to see if it had gained or lost popularity in the past couple of years. The graph has an upward trend, but without knowing what the trend for total volume on HN is, that information is useless.

http://hntrends.jerodsanto.net/?q=bitcoin%2C+groupon%2C+bubb...

XML vs Json http://hntrends.jerodsanto.net/?q=xml%2C+json that's close

Is it normalized against the growth of the website?

Excellent work! http://hntrends.jerodsanto.net/?q=excellent

vim/emacs http://hntrends.jerodsanto.net/?q=vim%2C+emacs

Awesome tool, but just a heads up: I tried searching for "C" and "C++", and it gave me the exact same line each time.

Is there any way to normalize by number of users? (Or am I missing something?)

This is really great!

http://hntrends.jerodsanto.net/?q=bitcoin

Looks like a bubble...

this is awesome, it would be a dream if advanced queries could be comparable though

this is great, awesome how its so fast.

slick!

Very cool.

Amazingly, C# is talked about precisely as often as C++!

http://hntrends.jerodsanto.net/?q=c%2B%2B%2C+c%23

;-)

I think it may ignore the #, ++, along with various other symbols. See this link (you may have to refresh once or twice if it doesn't work the first time): http://hntrends.jerodsanto.net/?q=c%2B%2B%2C+c%23%2C+c-%2C+c...

Lol. According to HN, Chrome should be the world's top browser. http://hntrends.jerodsanto.net/?q=safari%2C+firefox%2C+inter... -or- http://hntrends.jerodsanto.net/?q=safari%2C+firefox%2C+IE%2C... (with "IE")

Great site by the way. Nice job.

From where HNSearch is pulling data? Do we have any dump for HN?

Reply on news.ycombinator.com