Hacker News Reader

Hacker News

Comment by seabass-labrax | Hacker News Reader

Parent Full thread

seabass-labrax

I believe Teddit uses scraping; I run a Teddit instance for myself and haven't needed to set up an API key.

View on HN

From a very brief skim it doesn't appear to scrape: https://codeberg.org/teddit/teddit/src/branch/main/routes/ho...

But I'm not really a web dev so I might be misreading things.

Also: https://codeberg.org/teddit/teddit/issues/400

You can (currently?) add .json to any url on reddit and it'll provide a json object.

- https://codeberg.org/teddit/teddit/src/commit/b0ce6c52a66987...

That's another feature that that seems on its way out, like locking up the api..

Do you mean that they've said it or just makes sense?

Curious if they've said anything about it.

The json thing is an API request, so it will come under the new API rate limits.

It would be great for someone to scrape Reddit and expose that information in a format compatible with the official API.

So if you call /get-comments/1234 it scrapes post 1234 and returns the JSON object exactly as the official API does.

Then third party clients can just point to this endpoint.

Won’t they just get rate limited or hit with captchas

Currently, just add .json to the end of the url.

I have no idea what will happen to this with the changes.

If what you suggest is done, even a package, we could probably do a distributed pushshift[1] alternative to aggregate the data like the archiveteam's warrior[2] does and keep publishing the monthly data.

[1] - reddit.com/r/pushshift

[2] - https://wiki.archiveteam.org/index.php?title=ArchiveTeam_War...

Right now it seems legal to scrape Reddit. But given their trajectory of making the API fairly expensive to use, do you think it's likely that they would also limit/prohibit scraping (assuming apps like Apollo start scraping as an alternative)?

My understanding is that scraping of public websites is generally legal, isn't it?

Legal, but probably against ToS

That ToS is meaningless if you scrape logged out.

Reply on news.ycombinator.com