Hacker News RSS Feed with Full Text Articles | Hacker News Reader

Hacker News RSS Feed with Full Text Articles

nirmalpatel.com

32 pointstalison23 comments

I love that you did this. Good work. I intend to use it, and look at the code as well.

It does raise ethical concerns. Obviously, the folks aren't selling any advertising, if we're accessing their site content another way...

Which, as you correctly state, is exactly what RSS already allows. I've never understood why people put RSS feeds on their blogs. It's insanity. Give people a way to NOT come to my site and view my ads. Fine if ads are not your model, but for 95% of blogs, ads are the model. RSS is a fad, and eventually, saner heads will prevail.

Until that time comes, as you say, you're presenting the same info that an RSS feed does, but with the benefit of having been ranked and vetted by the HN community first.

RSS may go away, but something better will take its place.

People put RSS on their blogs for multiple reasons. I use it on mine because it greatly increases the usability of my site. Another way it helps is that it makes sites "stickier"; without RSS, people would not have something automatically telling them "Hey, that site you liked updated! Go check it out!". They would have to either remember the URL, or remember a link path to the site, or bookmark it, and they'd have to periodically check back. (Not that I particularly care—while it is nice to have readers, I write mainly for my own amusement, and my site is not commercialized at all.)

And I think that these two factors are a net win for most sites, whether commercial or not. There are ways around the advertising issue—offer truncated feeds, put ads in your feeds, provoke comments, etc. Plus, I thought that the known wisdom was that regular readers don't use ads much, it's the ones that stumble onto your site from Google, etc.? I haven't seen a study, though.

Also, for the record, I basically don't check sites that don't offer RSS. If it's particularly good, it'll get bookmarked, and maybe I'll come back to it in a few months when I've got some free time. But most sites don't even get bookmarked. On the other hand, if I find a site that had a good article, I'll often add it to my RSS reader on a trial basis.

Counting the seconds until someone flings a takedown order. Aggregating full content without permission definitely exceeds the bounds of "fair use".

It's been working at least for a few months without getting shut down.

So it's under the radar. That doesn't change the fact that aggregating full content without permission exceeds the bounds of "fair use".

How is this different from their own RSS feed? I think it's just saving them bandwidth.

It might not feature their advertising, their FeedFlare (if on FeedBurner - a great way to drive people to comments sections), special formatting, or tracking graphics that they use for statistics.

Before the inevitable "But I don't care!" responses, whether or not these things are interesting to us as readers is not relevant - they're still differences that could cause publishers to be unhappy and cry foul.

On the plus side, since the HN news is just an aggregation of many sources, it could encourage people to find new sources and subscribe to them directly in the long run. As a content provider myself, I'd be happy with this aggregator if it was indicated in each post what the original source is.. if that's missing, it needs to be added. We want attribution even if we're being syndicated ;-)

I agree completely that the service is infringing on the intellectual property of the authors of stories that HN links to. I just don't think that it is likely to get shut down, because, as you said, it is under the radar.

By whom? Nirmal, who created the tool, but whose work could be easily copied? Google, who have no control over which feeds people choose to subscribe to? Me, for just installing it?

Actually, I might well prefer just the first paragraph and/or the top 3 three comments - I don't know yet whether I want the full articles. However, I don't think it's as cut-and-dried as you say.

These debates will continue until online publishers come up with a convenient and affordable micropayment system. I'm not opposed to advertising per se, but I am against visual the clutter it often entails. And sorry to HN designers, but the same applies to a lot of websites. Less is more.

"By whom?"

Obviously by anyone hosting the feed for republishing full content without copy permission. I'm not happy with the current trend in copyright law, either, but it is was it is.

"I don't think it's as cut-and-dried as you say."

It really is.

Edit: your downvotes and personal opinions won't stop you from getting takedown notices and losing lawsuits. Ignore the legal reality at your own peril.

Obviously by Nirmal [...]

Really? But Nirmal has no way of knowing what articles people will see fit to post to HN. How can he negotiate permission with all possible content providers, given that I may link tomorrow to a brand new blog created later today?

I question whether he scan truly be said to be publishing, when it appears that the feed in question simply reformats without asserting any ownership over the content or monetizing it in any way, and involves no editorial selection on his part, something I consider a defining feature of publication.

"Really?"

Yes, really. And it's necessary to have a more thorough understanding of the current issues in copyright law and recognize that it's based on statutes and case law before speculating about what may or may not violate fair use. Words also already have definitions (eg, http://www.answers.com/publication and http://www.answers.com/publish) and, in a legal setting, those are the definitions you'll need to work with whether or not you have your own. I realize that this sounds somewhat ungracious (sorry! I mean no harm), but it's a legal issue, not a purely speculative philosophical issue.

I'm a little perplexed that you'd refer me to the dictionary definition rather than one from statute. This is the satutory definition of publication:

Publication is the distribution of copies or phonorecords of a work to the public by sale or other transfer of ownership, or by rental, lease, or lending. The offering to distribute copies or phonorecords to a group of persons for purposes of further distribution, public performance, or public display constitutes publication. A public performance or display of a work does not of itself constitute publication.

It was this definition I had in mind when questioning whether Nirmal's feed meets the legal threshold for publication. I'm not considering his work exempt under 'fair use' tests, but rather because Nirmal's involvement here is limited to disintermediating the gap between feed reader software and end content. The lack of monetarization by Nirmal has legal significance, as does the unavailability of the content in human-readable form on his website.

"This is the satutory definition of publication"

I don't see where it discusses "editorial selection" which you consider to be "a defining feature of publication"

"disintermediating the gap between feed reader software and end content."

Republishing content under copyright with XML tags and an .rss extension is no different than republishing content under copyright with HTML tags and an .html extension.

"lack of monetarization by Nirmal has legal significance"

Just because a website doesn't have ads doesn't mean it's suddenly immune to copyright law. It doesn't pass any of the fair use tests, and even if it did, courts are notorious for siding with copyright holders.

"as does the unavailability of the content in human-readable form on his website"

It's just as human-readable as HTML, both raw and rendered, and modern browsers (and all RSS readers) render it just as they do HTML.

As I said, I am opposed to restrictive copyright and am devoted to CC, but copyright is what it is. If you disagree, feel free to publish a feed of full Associated Press articles, bring it to their attention and see what happens.

Obviously [...]

That's only if he shows up on their radar, which well, he probably won't. And if he does you just take the source he posted and run it on your local machine, which presumably would fall under fair use.

For what it's worth I agree with you, but that also makes me wonder how it is that scribd can copy an entire copyrighted article without asking permission, and then say that to take it down they require a request that "meets all criteria for validity as a DMCA affidavit."

That seems wrong to me. If it's copyrighted, and they copy it the entire thing without permission, how is that not wrong?

EDIT: I see this has been discussed before many times, but this time I ask - scribd are apparently effectively immune because users upload things to them. They then claim that if it's copyrighted, it's not their fault. HN has uploaded content to them without the author's permission. Does that make HN a copyright violator?

HN only publishes links and, often but not always, headlines. Most publishers are OK with this, but not all. Recently the AP has reaffirmed that they consider copying headlines to be copyright violations and are threatening to again go after sites (notably google).

IANAL, but here's my understanding of the DMCA's Safe Harbour provision, which is why scribd and other UGC sites (e.g. YouTube, isohunt, flickr) aren't held responsible for copyrighted material appearing on their sites. UGC websites can be protected by the Safe Harbour provision under the following conditions:

- The site provides a legal, legitimate service (i.e. fair use)

- The amount of UGC on the site is too much to reasonably police

- They have an employee registered with the US govt as the agent for handling all copyright claims. Simply providing an email address, contact form, or even a mailing address doesn't count.

- When they receive legitimate removal requests from the copyright holders they remove the content in a timely manner.

As long as these conditions are met they can't be held liable for copyright violations which occur through the use of their product.

HN is a different case, because the amount of content copied from the source site (the headline, and only in some cases) is a small portion of the original work. Furthermore, HN is adding value to the content through ranking and community, which makes it a distinct piece of intellectual property. Similarly, you can legally make a collage which contains copyrighted material because the collage itself is a distinct entity (more than the sum of its parts).

How much of the page content can scraped and presented, or how large the pieces of a collage can be, is vividly debated. For me personally, its a bit of a Potter Stewart situation, and this feed is absolutely copyright infringement.

I didn't realize this had been submitted before (by nirmal). It was hosted on a different domain in the previous submission which is why this wasn't caught as a dupe.

Anyway, you can see some of the author's comments in this thread:

http://news.ycombinator.com/item?id=542334

I brought up some concerns about full content scraping last time this was discussed: http://news.ycombinator.com/item?id=542552

I used the feed for a few months but found the feed-readerized articles lifeless, frozen in time and free of discussion. I'm also not a fan of the title-only RSS feed. Some sites simply lose too much in feed form imo.

I love it! I have been looking for a RSS feed like this for quite a while now. Good Job!

Thanks, I was looking for this the other day!

Either it is a bug in Firefox 3.5 RC or the mime-type provided is wrong. It just renders like text, and not like an RSS-feed I can just click to subscribe to.

Since RSS feeds seems to work on other sites, I'll assume the problem is with the feed. Correct mimetype should be application/rss+xml.

Confirmed:

   $ wget -O /dev/null http://nirmalpatel.com/fcgi/hn_feed.fcgi
   --2009-06-22 21:01:56--  http://nirmalpatel.com/fcgi/hn_feed.fcgi
   Resolving nirmalpatel.com... 207.210.105.86
   Connecting to nirmalpatel.com|207.210.105.86|:80... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: 211754 (207K) [text/plain]
   Saving to: `/dev/null'
   
   100%[==================================================>] 211,754      254K/s   in 0.8s
   
   2009-06-22 21:01:57 (254 KB/s) - `/dev/null' saved [211754/211754]

Reply on news.ycombinator.com