Hacker News Reader

Comment by damnesian | Hacker News Reader

>RSS feeds make web scraping and content stealing easier. This is a legitimate concern.

This is a negative way of looking at the reality- rarely mentioned in articles about RSS: RSS may be "dead-" however, there are many mobile apps that are fed by RSS feeds. Having one may risk some loss of revenue- but you'll lose even more by simply being omitted from these aggregator apps, all of which (i mean, ok.. probably all) link back to the source website. That's a huge potential for added exposure and revenue.

View on HN

I don't get the argument at all. If someone is so averse to publishing a feed of full article text, they can still publish a feed of headlines, first paragraphs, and "read more." It's tedious to read those but much better than no feed.

I actually like it when feeds have some sort of TLDR in their feed body. It works as a quick way to determine if I want to read the article or can skip it.

Fowler's blog does this (or maybe Feedly does?) but not many others, afaik

Now that I think of it: that would a very nice use of an LLM: some proxy where I put an RSS into, and that publishes a modified RSS feed where the <description> is then replaced with a very short summary of the original description.

I include RSS feeds specifically because I want make it easy for website users to use my site, even for scrappers (and independent search engines, and AI training, ..etc). The idea of "stealing" content doesn't even make sense. How does one even steal information.

People who think like this need to get over themselves.

I live from my website. Stolen content uses my labour but routes its fruits to someone else who adds no value. I'm competing against copies of my own work.

Worst of all, the lazily copied versions of my work introduce serious errors because the authors don't know what they're talking about.

Oh and some copies are used for phishing scams, so that's another concern.

>Stolen content uses my labour but routes its fruits to someone else who adds no value.

This has nothing to do with "web scraping". Scraping means downloading. What you are complaining about is copyright infringement: they are downloading your web pages and re-uploading them elsewhere. That's completely unrelated.

They're the same. It's just a matter of whether a machine or a human does the paraphrasing. With AI, the latter will become very common.

Downloading and uploading are not the same. 'Scraping' a website is just what all of us do every day.

I understand this issue and really feel sorry for anyone who has to deal with this. I just don't understand how explicitly not implementing RSS would even her with this [not saying you specifically claim this]

It enables machine-stealing. The sort where the whole content gets copied and hosted on a content farm. This is admittedly a smaller threat as those rank very poorly in the search results. I was mainly answering directly to the parent comment who doesn't understand why people don't like their work to be stolen.

This is what stops me from open sourcing the whole website, since it's just Markdown. It would be good to open it up to contributions, but the website is already copied enough as it is.

> It enables machine-stealing. The sort where the whole content gets copied and hosted on a content farm.

I understand that but lack of RSS is not going to prevent that. It's barely a roadblock for those who steal.

Having the web pages accessible publicly are what makes machine stealing possible, rss has nothing to do with that.

> It enables machine-stealing.

Publishing your website at all does that. Feeds are not at all required to scrape websites.

My personal favourite: someone stealing a random post, backdating it to a couple of days before I published it, and then sending a DMCA to my hosting provider for my own damn article.

Does that happen? Hot damn. I never considered this attack vector. It's brilliantly evil.

That sounds made up but they don't need an RSS feed to do that. RSS could actually help with your defense because it means others could have a copy of your post from your feed fetched before the liar's.

It's incredibly common to only publish a headline and summary to RSS and then have people click through to your site for the full article, so it seems like you're blaming RSS for what is a user error

This lessens the payload of the RSS as well not shipping dozens of whole articles.

This is what I do. I have RSS feeds on all my websites, but I don't put the whole article in them. This works well enough for my needs.

> People who think like this need to get over themselves.

So it's fine if someone took the fruits of your labor and sold it without permission?

If its in a rss its not called web scraping...

I'm the author of the linked post and I agree with both of you. But this is an objection I got on an earlier draft of the post, so I mentioned it.

And it assumes you’re blogging for money, which is not universally the case. If you’re not blogging for money, those objections are of no value.

Does RSS make web scraping easier, or does it make it so you can avoid doing web scraping altogether? Web scraping is hard to stop, but if people just want to get the content, an RSS feed probably uses less resources for everybody compared to web scraping.

This would effectively remove you from search engine indexes as well. To fix that, you would have to add that URL to robots.txt, but any decent scraping tool would also pay attention to that so you’d be back to square one.

You can probably find or pre-emptively exclude search engine indicies IPs from the blocklist. It isn't like there are more than a few out there, sadly.

> This is a legitimate concern.

This isn't even true. If you publish information on the web, then people are going to download it to their computers. That's the whole point of providing it. When someone accesses your website, your web server actively responds with the web page.

Complaining about people doing this is like complaining that someone asked you for money and then you gave it to them. Yeah, they asked, and you gave it to them? If you don't like it, don't give it to them.

In other words, if you don't like people scraping your website, don't give them your web pages when they ask for them.

Reply on news.ycombinator.com