Ask HN: How do I scrape my tweets without using Twitter's own tool?

4 pointsmatteocapucci2 comments

Given the state of the Twitter workforce I suspect my request for data wouldn't be fulfilled anytime soon. Yet, there must be a way to scrape my tweet history just using the APIs. Does anyone know of any good (and ideally free) online service I can use for this?

View on HN

You can use Tweepy (Python Package) to query their API for the data. You will need a Twitter Developer Account but the setup is quite simple. You can use the instructions at: https://towardsdatascience.com/how-to-access-data-from-the-t... to set it up.

I wanted a client to export my followers, Following and Lists (Members) as the official export didn't have it. Used Tweepy to create a script in a couple of hours to pull this data. It should it fairly simple to modify it to pull the tweets as well if you are interested. Or I can take a look and add it when I get some free time.

https://github.com/suramyatomar/TwitterExtract

I've recently discovered on a substack about web scraping the existence of anti detect browsers. Using Selenium or Playwright you can automate these browsers, each can have hundreds of isolated profiles, with different fingerprints and cookies. If you add also a proxy layer, you should be ok.

Reply on news.ycombinator.com