Show HN: Page and crawler that show flagged HN submissions only | Hacker News Reader

Show HN: Page and crawler that show flagged HN submissions only

mosermichael.github.io

5 pointsmm3219 comments

Wow! I did cause them to change: now they flag stories, but they also delete the title from the story, like here: https://news.ycombinator.com/item?id=29134228 . Now you only see [flagged] in the title. That makes my little exercise here much less useful. That's also a lesson: crawlers can be made ineffective by slight changes in the input. At least i had some fun at making this little exercise...

On the other side: it makes this site a bit less interesting. I am sorry for having caused this, i had the best of intentions.

Now i need to find another toy to tinker with. (My wife says that all of my projects are essentially useless, as they don't yield much in terms of money. Well, she has a point... :-( ) Here is a list of my projects: https://github.com/MoserMichael

Nah, they had that already. Just got it after scanning all of the entry id's by range.

Here is the crawler project: https://github.com/MoserMichael/flagged-hn/ . It crawls HN and creates a database record for each submission. The page is generated by formatting the result of the sql query. A bit boring, all written in python3.

(I don't have the nerve to run the crawler and page generator on a daily basis; the page has instructions on how to install and run the script)

Is the Hacker News API not useful for your project?

didn't know about it. https://github.com/HackerNews/API . Checked it out, the api wouldn't have helped me: the API reponse for this flagged story doesn't indicate, that it has been flagged. https://hacker-news.firebaseio.com/v0/item/29103056.json?pri...

On the other hand: they gave me an idea. I can structure the crawler differently. Right now, I am now following the next page (apparently there is a limit on the number of 'next' pages). Instead, it is possible to get the maximum item id, then just decrement the item id down and down again.

for this task I need to rescan the pages often, in order to check if the post got flagged. There is no way to know, if or when they rescan the submissions with the HN api.

Thanks for your responses.

So you have to check the HTML (substring search) to see if a story is flagged?

Edit: Answered my own question: yes: https://github.com/MoserMichael/flagged-hn/blob/master/crawl...

Thanks for your feedback, it is very much appreciated!

yes, it's part of the table row that includes the title, but if you take the text within the link tag only, then you don't get that indication - I think that's what the HN api might be doing.

Example:

   <td class="title"> [flagged] <a href="https://www.dailywire.com/news/mcauliffe-education-plan-could-lead-to-1970s-style-busing" class="titlelink">McAuliffe Plan: Giving Parents a ‘Choice’ Leads to ‘Segregation’</a><span class="sitebit comhead"> (<a href="from?site=dailywire.com"><span class="sitestr">dailywire.com</span></a>)</span></td>

Actually if you ask me, I don't know why they are using this 'flagging' thing over here, they might have just moved a submission some pages away from the front pages, if they don't like it.

Users with enough Karma can 'vouch' for a flagged entry, and effectively de-flag it if they think the flagging is erroneous. Flagging isn't a one-way street.

Reply on news.ycombinator.com