Ask HN: Are there any web/text scrapers that work with Facebook Groups?
There are lots of gems and goodies hidden in there.
The search system for Facebook Groups is dismal.
I have so far found no solution that works.
This is not just a wish for me or the other admins but a big wish from the community as well.
a) Various products / systems based on scraping groups but only to (maliciously) scrape names, emails and whatever other contact information can be had.
That does not help me at all.
I do need to know the nick of the user, but I want is the actual content of discussions.
Further these products seem somewhat shady to me.
I dont really want some 3rd party to run it from their end either.
b) I have not been able to find enough of a an example of writing one. I have found useful content on Python but I am missing some crucial pieces.
Like how do I run it in the context of my logged in browser (?)
I would like to do this as close a regular person browsing as possible to avoid FB taking offense.
c) I have tried some Macro plug ins for Firefox but I cant get it to understand the recursive nature of threads.
Find the next thread is not hard, getting the first content is not hard,. but telling it to "keep clicking" all the way down is not something I have mastered yet.
d) I have tried doing it myself, little by little, copy and pasting, but it is soul crushing work and the body of content is huge.
I would prefer something I can run myself. I would prefer it to be slow. Presumably robot fast descent into every thread my wakeup FB bots?
I really wish FB would allow admins (with users voting for it) to download the content of their group to keep as a backup.
Are there any seasoned scrapers out there who can help me?