Hacker News Reader

Comment by stared | Hacker News Reader

Author here.

I wrote a description in the main file, so I don't have to add much here. Though, I am open to feedback (especially for the graphical presentation, as the taxonomy & subreddit list is not mine).

Even though I like and use D3.js a lot, it is my first post using ObservableHQ. It seemed to me to be a nice choice for this kind of content. Considered bl.ocks.org or repo/GH-page, but I think ObservableHQ makes it easier to edit and share.

View on HN

This is neat, however (here comes the harsh critic) :

- Rendering a simple tree like this would work just as well, if better, in text only format (like the output of the "tree" command on linux) : https://justpaste.it/6r9un

- The classification also seems a bit manual and arbitrary.

Maybe giving each node a weight would make d3 worth using ? Maybe scrap the subscribers of each subreddit, and make a force directed "map of reddit nsfw" ?

Your skills seem wasted on this if you're good at d3

Your text-based tree is nice, but it comes with pros and cons, as every datq visualization. In fact, it was my first idea, but I felt I need to create something better for exploring the high-level structure.

But feel free to do a fork and a viz in this line: https://codepen.io/brendandougan/pen/PpEzRp

There are quite a few maps of Reddit according to co-participation in subreddits.

Though it does not make it semantic. (Maybe semantic things can be extracted from the post content; though, I didn’t try.)

If you wanted to do more than just a straight tree: For any pair of subreddits draw a line between the subs weighted according to the number of commenters that commented in both subreddits. Maybe doing a similar graph with people that post in both subreddits would be interesting too. I expect both variants to yield a somewhat more natural grouping of subreddits than an artifically curated list, but making it readable and explorable would be a challenge.

0: http://files.pushshift.io/reddit/comments/

Well, if you want to do it, go for it.

If I were to do so, I know more suitable methods, vide description from https://github.com/stared/tagoverflow/. Without such normalization, it won't work given that there are networking with different orders of magnitude of users.

Right now don't have much time for this project, and well - I am this kind of person that has a few dozen times for idea than time, not the other way. :)

> The classification also seems a bit manual and arbitrary.

That's because it was basically copied from /r/NSFW411's wiki.

Thank you a lot, a mystery solved!

https://www.reddit.com/r/NSFW411/wiki/index

This is very interesting! I was wondering what the lack of more LGBT (specifically gay) subreddits is due to and why they are grouped the way they are? What I mean is I know there are are a number more gay-related subreddits and I found it interesting that while straight porn got broken down into it's categories all LGBT porn was lumped together under high-level categories (ie. gay/lesbian/transgender/etc).

Edit: Look like I misunderstood what "nsfw_data_source_urls" really was (I thought it was a list of urls to specific images, not a list of subs broken down like it is). I thought it was used to train not the source of the categorization. Given that I understand the grouping more.

It would be great if you could overlay the number of subscribers of a subreddit in it. It would be a neat way to see what is most prominent. Also, and I don't know if this information is possible to get, if you could get intersections of users in different subreddits, you could see a graph of which subreddits are most tightly connected.

There are such projects, and in fact - I mention two of them.

If you want to obtain data, Related subreddits shared it (150GB or so): https://github.com/anvaka/sayit

How did you get the URLs given the reddit history search limit? Was it from the big-query dataset?

To be honest, I have no idea. See https://github.com/EBazarov/nsfw_data_source_urls/ for the data source I used.

There is a Reddit bulk downloader capable of downloading ~1000 images from a subreddot: https://github.com/aliparlakci/bulk-downloader-for-reddit (or just image links, if needed); I tested it and it works well.

Though, I have no idea if it was used for the project.

Reply on news.ycombinator.com