Show HN: Hacker News Trends | Hacker News Reader

Show HN: Hacker News Trends

eliot-jones.com

137 pointsUglyToad68 comments

Some more background on what this is and why I built it here: https://eliot-jones.com/2020/1/hackernews-trends And the source code here: https://github.com/EliotJones/HnTrends

As I mention in the blog post this is hosted on the lowest tier DO droplet so will probably fall over soon.

Have you considered deploying to Google Cloud Run or something like that which charge you per-request rather than instance-hours?

That's a good idea. It's currently on the box I run my blog off so shouldn't be costing my anything at all (given I am paying for the blog hosting anyway) but it would be interesting to try.

FWIW your cache appears to be case sensitive but searches aren't.

Good spot, I'll fix it.

This is super cool and potentially really useful (and thanks for building it!)

One q - I can't seem to click through to a specific date? Is this me missing something or you not making it available? Would seem the next obvious interaction?

Thanks, yeah this functionality doesn't exist at the moment (aggregation happens server side so the actual stories aren't available client side). You can however get the full data for a trend from the API endpoint (the front end doesn't use this endpoint but I added it since it's useful to check the actual data). Just replace 'rust' at the end with your search term: http://eliot-jones.com:5690/api/results/rust

Frontend framework comparisons:

React: http://eliot-jones.com:5690/Home/Trend?id=React&allwords=tru...

Angular: http://eliot-jones.com:5690/Home/Trend?id=Angular&allwords=t...

Vue: http://eliot-jones.com:5690/Home/Trend?id=Vue&allwords=true

From a purely dataviz standpoint, this is a fail. The optimal visualization would allow for all 3 terms to be shown on the same graph, or at least the same page. 3 distinct graphs could be a decent comparison, if the y-axis remained constant. But since it doesn't, every graph peaks at the same approximate height, so really the only thing an end-user, seeking a comparison of similar concepts/terms, would likely find useful is the max displayed value on the y-axis.

I recognize this is little more than a proof-of-concept at this point, so perhaps "fail" is a strong word, but hey, it's what the author chose to submit.

This is a fair criticism and from the feedback represents a fairly major missing functionality in the site. For what it's worth I only realised this might be a useful feature after I finished version 1. From my write up[0]: "Allow multiple trends on the same plot. It would be interesting to compare Kubernetes with Docker directly, or PyTorch with Tensorflow to get an idea of the relative trends in attention".

Initially I built the front end to analyse a single term (recession) rather than for trend comparison but it would definitely be useful and I'll look at adding it.

[0]: https://eliot-jones.com/2020/1/hackernews-trends

I've pushed an update that allows you to display multiple trends on the same plot now, if you wanted to take a look.

For the monthly view, it should probably truncate the ending date so as to not show a partial month number. February just started so it makes every search term look like it is plunging.

I noticed this as well, it really confuse me for the first several searches.

Yeah it's annoying, I think I'll add a toggle "Display data for incomplete period" or similar so the user can switch between views (for something like "coronavirus" the data from the first few days of the month exceeds that from the previous full month so I don't want to discard it entirely).

A simple linear projection could be a good additional option too.

Slow, steady upward trend for racket: http://eliot-jones.com:5690/Home/Trend?id=racket

Would be nice, though, to compare relative to an evergreen term like "computer" the way stock performance gets compared to an index.

I assume the amount of HN articles per day has increased over the years since 2007 along with the growth of HN popularity. So, when I see a trend in some subject would it take that into account? Otherwise it could easily not be a trend but an increased amount of HN users.

Just curious before I get excited.

Great question. I've had an issue with this with a tool I built myself. I hope it takes it into account!

Edit: Just noticed this in a blog post from the creator:

> Since there are more posts per-day nowadays it's also useful to look at the count as a percentage of all posts in the period.

Yeah, peaks in story count nowadays are relatively smaller than peaks from 2007-2009ish. You can use the "Display as % of stories" button which is right of the "Group data by" drop-down to show the data as a % of daily/weekly/monthly stories posted to the site.

HN Bitcoin indicator http://eliot-jones.com:5690/Home/Trend?id=BITCOIN&allwords=t...

Interesting searches: Bitcoin (2 peaks, now down) Drone (up and down) Machine learning (post peak) AI (post peak with a sudden new peak) SaaS (steady rise to steady high, then fall and return) Dropbox - rise and gradual fall Quora - sharp start, low gradual fall Kickstarter - stunning fall over time

and so on. These & Uber, Slack, Tesla and other company names give a good picture of how 'exciting' these companies are to this particular market of tech early adopters, where exciting is defined as either new or controversial, which is the very definition of "news".

Neat:

http://eliot-jones.com:5690/Home/Trend?id=APL&allwords=true

http://eliot-jones.com:5690/Home/Trend?id=k&allwords=true

http://eliot-jones.com:5690/Home/Trend?id=arrays&allwords=tr...

It's now responding with

    An error occurred during a connection to eliot-jones.com:5690. SSL received a record that exceeded the maximum permissible length.
    
    Error code: SSL_ERROR_RX_RECORD_TOO_LONG

It's been doing that to me right from the beginning[0] and I've been bewildered as to how people are reading it at all[1].

No one has answered that question, and I've even been downvoted for asking it. I've given up.

[0] https://news.ycombinator.com/item?id=22237285

[1] https://news.ycombinator.com/item?id=22245516

Nice. One thought I had is it would be cool to be able to click on any (non-zero) point on the graph and see a list of the articles from that day, sort of a drill-down on WHY was that term peaking in popularity then?

That was my instinctive response when I made my first search; would be a great addition.

That's a good idea, thanks. It might challenge my limited JavaScript but it's definitely worth adding. At the moment you can get the full data from a graph using the API endpoint. For Tensorflow: http://eliot-jones.com:5690/api/results/tensorflow

Something I found a bit interesting (via trends)[0], did a search for "coronavirus" and there were a few small bumps several years ago[1]

[0] http://eliot-jones.com:5690/Home/Trend?id=coronavirus&allwor...

[1] https://hn.algolia.com/?dateRange=all&page=23&prefix=true&qu...

This is really cool. Seems the backend is a bit slowed down with the traffic. Would be cool if the graph had more horizontal space on a small phone without rotating it and re running the query.

Shouldn't you provide as a baseline the absolute number of posts? I tried three different searches and all showed monotonic growth.

You can use the "Display as % of stories" button just above the results plot which should account for the growth in the total submissions per day, or did you mean something else?

I meant exactly that. Thanks!

(I apologize, I cannot edit my previous comment from the client I'm currently using.) I'm not seeing that % option, though. I'd be happy to send you a screenshot somehow, if it helps.

That is odd sorry about that, you can email it to me "elioty at hotmail dot co dot uk" or stick it in a github issue: https://github.com/EliotJones/HnTrends

   504 Gateway Time-out

When searching for "Google". I assume the number of mentions is too high.

For those interested, if you'd like to compare trends (by sentiment or volume), I built a similar application.

https://hnprofile.com/compare?search=AWS,GCP%20|%20Google%20...

Hi, this is cool but you have hijacked the back button!

the max timeline doesn't work for me. Chrome or FFox

Robotics/robot: http://eliot-jones.com:5690/Home/Trend?id=robotics%20robot&a...

Potentially interpreted as interest peaking 2015-2019, currently in decline.

I was randomly thinking about that a few days ago, seems to me like robotics is not seen as the "future" as it was before, that's a shame.

I think the potential for home robots is huge, but people probably now realize they are also very dangerous because of all the information they could gather on individuals.

I guess I'll wait for selfhosted decentralized robots then :(

Here is a sarcastic view that has elements of truth.

The old sales is model is consumers owned houses. The could then buy an 'appliance' to last many years for goodly-portion-of-monthly-or-annual-wage, on the promise it will 'save you time' and 'improve quality of life'.

The new model is consumers rent forever. They buy an 'experience' to last a second for ripoff-price-but-compares-well-to-drinking-or-theater-tickets, on the premise it will keep them unfocused and temporarily fulfil their deeply vacuous lives, thus staving off an expensive shrink or a higher does prescription of antidepressants.

In this new model, robots will not be owned by consumers, but will be public service infrastructure. This is because the consumers have nowhere to put them (don't own a home), move too often to justify purchase, and cannot afford a substantial outlay anyway.

Every time I try to reach this site I get "This site can’t be reached, eliot-jones.com took too long to respond." (ERR_CONNECTION_TIMED_OUT). Going to the website without the port works, but apparently the port is necessary to reach the trends page. Chrome(OS) 79.

It would be illustrative to show the rate (%) as an alternate to the absolute number of posts.

This should be possible using the "Display as % of stories" button, or does that do something different to what you want?

That's exactly it. I missed it.

Neat! Feature request: Let us graph a specific HN member's submission count per day.

http://eliot-jones.com:5690/Home/Trend?id=--&allwords=true

Searching for "--" results in an Error page.

Thanks for the bug report, I should probably add some front end validation for the search term since in the current design Lucene will only index letters and numbers (which is a problem for searching languages like C++).

How are people reading this? When I try to load it in either Chrome or Firefox I get "502 Bad Gateway" and "Secure Connection Failed" respectively.

I assume people are somehow accessing this ... how?

the last data point is a little irritating.

maybe disregarding or scaling the last data point for the whole month would make the graph smoother

also, i am really enjoy it

it would be nice to add multiple terms for a search to compare them in the chart

Also was trying to compare multiple terms.

Thanks, yeah I agree, the percent view smooths this out a bit but it still gives a false impression. I've been considering various different solutions to it today but I've not settled on the correct solution yet given the data as currently presented is 'correct' but not particularly useful.

HN seems to be much more interested in Trump than it ever was in Obama:

http://eliot-jones.com:5690/Home/Trend?id=obama&allwords=tru... http://eliot-jones.com:5690/Home/Trend?id=trump&allwords=tru...

"NSA" keyword gives an interesting result

Hacker News: Hyperventilating about Rust Since 2010.

The link is dead for me. Anyone else?

It should be back now, though you might be running into the same SSL issue ColinWright was elsewhere in the thread.

great work! Is it possible to open source the 2.8 Million Hacker News posts database?

I definitely want to but I'm not sure what the situation would be with copyright (or indeed if I'm allowed to use the posts for indexing in the way I have). I'll email HN and see what they say.

thanks, great to see the data!

I don't think Hacker News trends are representation of popularity worldwide.

Ah trends, the modern replacement for getting it.

    Secure Connection Failed

    An error occurred during a connection
    to eliot-jones.com:5690. SSL received
    a record that exceeded the maximum
    permissible length.  Error code:
    SSL_ERROR_RX_RECORD_TOO_LONG

    Please contact the web site owners to
    inform them of this problem.

Can't do that, this will have to do.

Does Firefox do this by default or is there a setting to upgrade http to https?

As mentioned briefly on the blog I didn't want to spend any money on this (currently not employed and living off savings) so it's sitting on the droplet also running the blog, I assume only 1 thing can run off 443? If that's the case I won't be looking to enable ssl for this, apologies.

I have no idea, but I assume that if one asks for https and is only offered http, that would be a security hole. I just tried asking for http and it reverted to https.

I can't help, I have no knowledge of these things. Sorry 8-(

Ah, OK, sorry to hear it doesn't work. It's strange, it shouldn't be attempting to use https (though for security reasons it should so I assume Ive been testing against outdated browsers). Sorry.

You should be able to make the webserver reverse proxy from https://eliot-jones.com/hntrends to http://localhost:5690, e.g. with nginx [0] and httpd [1]

[0]: https://docs.nginx.com/nginx/admin-guide/web-server/reverse-... [1]: https://httpd.apache.org/docs/2.4/howto/reverse_proxy.html

Reply on news.ycombinator.com