Minor stuff: Printing instead of logging. Would prefer a package that only does the retrieval and nothing else. Hardcoded SQL(ite?) storage.
Minor stuff: Printing instead of logging. Would prefer a package that only does the retrieval and nothing else. Hardcoded SQL(ite?) storage.
For Twitter in specific, isn't HTML scraping vastly preferable to using their official APIs? Otherwise you run into pretty arbitrary usage limits and missing features.
There's a small list of services where I think I prefer HTML scraping and browser piloting for a 3rd-party client: Twitter, Patreon, Facebook, LinkedIn, a few others. Services where the official APIs are underdeveloped or crippled to the point of almost uselessness.
Then it turned out Twitter refused half of the attendees the API key.. (maybe they thought it was spam coming from the same wifi, same time).
So then I just gave out my API key to the rest of the class, and in a few minutes it was blocked..
For a service that has a history of empowering users to protest and to spread news in crisis situations, it's a shame their API is so locked down.
The hardest part when working with data is often not manipulating data per se, but spending time on crap like this.
Took me 20 minutes to install twitter-dump, and despite a successful twitter-dump auth, I end up with 'Request exception Forbidden {"errors":[{"code":200,"message":"Forbidden."}]}'.
Not going to spend time to fix that, I'll use the dirty solution that works.
Maybe they could combine efforts? Maybe they could look at the code, and if the licenses allow, port things to theirs.