It runs in your browser like a browser extension. It scrapes the HTML and navigates from page to page.
Setup and usage instructions are in the file.
Check out https://gabrielsroka.github.io/getHNFavorites.js or to view the source code, see getHNFavorites.js on https://github.com/gabrielsroka/gabrielsroka.github.io
https://github.com/reactual/hacker-news-favorites-api
Here's an example query:
Would a client using your API paginate to fetch, say, 50 pages? When I tried it using ?limit=50, I got a 504 error.
Thanks!
(Edit, never mind, I see you explain it in the readme.)
Plug for a related golang tool I wrote and use which exports favorites upvotes as structured JSON:
https://github.com/jaytaylor/hn-utils
Just
go get github.com/jaytaylor/hn-utils/...I've been having some fun with the API recently building this tool: https://github.com/dogsheep/hacker-news-to-sqlite
Related Question: Is this the source code for HN? https://github.com/wting/hackernews
Not upvoted yet as they aren't in the API I am using.
That's great news! Is there a way to be notified (eg, via email) when this comes out?
Thanks.
I wanted to show people that it's possible (and easy) to get to your own data!
(Edit: haha, I see what you just did there. A little recursive humor.)
I used the jQuery selector `a.storylink`.
#!/usr/bin/env python3
import requests
from bs4 import BeautifulSoup
for p in range(1, 17):
r = requests.get(f'https://news.ycombinator.com/favorites?id=app4soft&p={p}')
s = BeautifulSoup(r.text, 'html.parser')
print([{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')])One more question: what is the best way stop it when it will reaches last page?
> for p in range(1, 17):
Actually p=17[0] is empty (as p=16 is maximum as for now).
Maybe, script should scrap pages from `1` to `infinity` UNTIL it detect next message on page[0]:
> app4soft hasn't added any favorite submissions yet.
A better way to solve it is to look at the `len()` of the results, and stop when it gets to 0:
p = 1
while True:
r = requests.get(f'https://news.ycombinator.com/favorites?id=app4soft&p={p}')
s = BeautifulSoup(r.text, 'html.parser')
faves = [{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')]
print(faves)
p += 1
if len(faves) == 0:
break path = 'favorites?id=app4soft'
while path:
r = requests.get('https://news.ycombinator.com/' + path)
s = BeautifulSoup(r.text, 'html.parser')
print([{'title': a.text, 'url': a['href']} for a in s.select('a.storylink')])
more = s.select_one('a.morelink')
path = more['href'] if more else NoneI'm not sure if it still works as it too relied on HTML scraping. Perhaps I should update it to support favorites too.
Edit: Whoa, it's been 4 years already. I believe HN didn't have favorite feature at the time. That's why I used upvoting as my bookmarks system and created a script to export that data.
Edit: I see from the other PR it's called `upvoted`.
Edit 2: I changed it to `upvoted` and now I get a 200 OK, but the code crashed right afterwards on `tree.cssselect()`.
Btw, good work on your JS solution. It's great that it just works without requiring any download or installation. :)
According to Article 3 of the GDPR, it applies to:
1. Processing that takes place in the context of processors and controllers that are in the Union, regardless of whether or not the processing itself takes place in the Union.
2. Processing the data of subjects who are in the Union by controllers or processors who are not in the Union if the processing is related to offering goods or services to such subjects in the Union or the processing is related to monitoring the behavior of such subjects that takes place in the Union.
I don't know how HN is structured, but I've not seen any indication that they are in the Union, so #1 probably does not apply.
#2 applies if they are doing processing related to "offering goods or services to such subjects in the Union" or "monitoring the behavior of such subjects that takes place in the Union".
One of the recitals elaborates on the first branch of that:
> In order to determine whether such a controller or processor is offering goods or services to data subjects who are in the Union, it should be ascertained whether it is apparent that the controller or processor envisages offering services to data subjects in one or more Member States in the Union. Whereas the mere accessibility of the controller’s, processor’s or an intermediary’s website in the Union, of an email address or of other contact details, or the use of a language generally used in the third country where the controller is established, is insufficient to ascertain such intention, factors such as the use of a language or a currency generally used in one or more Member States with the possibility of ordering goods and services in that other language, or the mentioning of customers or users who are in the Union, may make it apparent that the controller envisages offering goods or services to data subjects in the Union.
Does HN "envisage" offering services to people in the Union? Or are they a site that is merely accessible from the Union without envisaging offering services there?
There's a recital that elaborates on the second branch, too:
> In order to determine whether a processing activity can be considered to monitor the behaviour of data subjects, it should be ascertained whether natural persons are tracked on the internet including potential subsequent use of personal data processing techniques which consist of profiling a natural person, particularly in order to take decisions concerning her or him or for analysing or predicting her or his personal preferences, behaviours and attitudes.
Does the data HN stores about its users satisfy this? And if it does, is the behavior being monitored taking place in the Union?