Show HN: Download Twitter data without API keys | Hacker News Reader

Show HN: Download Twitter data without API keys

tweetfeast.com

57 pointschr15m43 comments

In April last year I started thinking about using Twitter in a smarter way. I wanted to do analytics on my tweets and find out more about people following me on Twitter. What kinds of things do people who follow me like and retweet? I decided to dig into the data and find out.

When I went to try and download Twitter data in raw form I found I quickly got bogged down writing API wrangling code and fiddling with API keys. I just wanted to crunch some data but here I was wrangling Twitter's API. This was such a frustrating experience it suddenly looked like an opportunity to me. Was there room for a product here? A product which does one simple thing: help people extract Twitter data painlessly without writing any code.

I did some research and discovered some tools that purported to do this, but none of them worked well for my use-case and all were badly designed and/or expensive. I decided to take a shot at it.

I worked on this as a side project in a "calm company" fashion. Each week for 26 weeks I would put aside one day to chip away at the features. I tried not to think about how the issue tracker was filling up more and more. Several times I pared back the feature set to try to really focus on the core use-case.

When my first user reached out and engaged I knew I might be onto something. I kept posting progress on Twitter and a few more people started to use it each week. Some of them came back, hinting at possible user retention. I hired a writer to write some articles to help with SEO and I kept working towards and MVP that I could use to test the market.

Finally the day arrived where all of the critical issues in my issue tracker were closed. That meant it was launch day. That was yesterday. So here I am releasing this on Hacker News to you, dear reader.

God speed little micro-SaaS, may the winds of fortune be at your back.

Open articleView on HN

What assurances can you give that you will be around in a year because I think this could get crippled easily regardless of how you are doing the fetching on the backend (scraping, api, nitter). I'd love something like this to not have to wrangle with the api myself but it's giving uneasy vibes for longevity.

Edit: I tried it, got the following for "tweets from a search result"

Error: Request failed with code 429

Request exceeds account’s current package request limits. Please upgrade your package and retry or contact Twitter about enterprise access.

So they really were trying to sell access to the content you could fetch with a twitter api key?

Yes that's correct. The value proposition of the site is a nice UI and access to Twitter data that does not require writing code or using a Twitter API key.

This will be gone in days if anyone uses it, it's clearly against the terms.

You are right, API access has been suspended for my app. Do you know which rule or term the app violates? I am not able to find it in any of the documentation I read.

Your uneasy vibes for longevity turned out to be prescient. Less than 24 hours after launch Twitter suspended the account and locked my app's API access.

I have appealed the suspension as I genuinely do not understand which rule the site violates, but I don't like the chances of getting it re-enabled.

Thank you for checking out the project and taking the time to write a comment, much appreciated.

I like the spirit of this, but the monetization strategy rubs me the wrong way. You're basically reselling a UI wrapper for the twitter API.

Nevertheless, good luck and I hope they don't shut you down too fast.

3... 2... 1... cease & desist

Yes, the account has been suspended. I have lodged an appeal as I do not understand which of their terms has been violated.

Yes, I am selling a UI wrapper for the Twitter API. Why does that rub you up the wrong way? Many people do not have the privilege of being able to write code to interface with the Twitter API. I thought these people might appreciate a way to get Twitter data from the API anyway. It turns out some of them did but unfortunately as you predicted, access was shut down quickly.

Colour me curious, but why do I need to sign in with twitter to use this? And if I do and you get c&d'd do I get blocked by twitter?

Edit: spelling

You need to be signed in as the app makes the data requests on your behalf. My personal account was logged in and I did not get blocked by Twitter when they suspended the app account, so it seems unlikely anybody else would be.

What is the advantage of your implementation versus Twint, which is free?

Twint's last commit was on March 2021.

From the Twint "install" documentation:

> pip3 install --user ...

> git clone --depth=1 ...

> Noticed a lot of people are having issues installing (including me). Please use the Dockerfile temporarily while I look into them.

Most people in the world don't know what git is, what a Dockerfile is or how to use the command line or Python. The advantage of my implementation is you don't have to know those things to get data out of the Twitter API. You can simply sign in with Twitter and you get a nice user interface for downloading data from the API without having to write a line of code.

You were banned for either one or both of the following things: 1. Violating II. Restrictions on Use of Licensed Materials of the Developers Policy => A => ) provide use of the Twitter API on a service bureau, rental or managed services basis or permit other individuals or entities to create links to the Twitter API or "frame" or "mirror" the Twitter API on any other server, or wireless or Internet-based device, or otherwise make available to a third party, any token, key, password or other login credentials to the Twitter API; 2. violating the security model of the Twitter API, because with your Service Twitter is no longer in control of who can access their data over the API.

Thank you very much for finding this. It seems the violation is this past specifically:

> provide use of the Twitter API on a service bureau, rental or managed services basis

> I just wanted to crunch some data but here I was wrangling Twitter's API.

> I worked on this as a side project in a "calm company" fashion. Each week for 26 weeks I would put aside one day to chip away at the features.

So you wanted to crunch some data...and instead of spending 3-4 hours setting up the API properly...you instead spent 26 days writing an unofficial hack?

It seems like what you really wanted was to spend 26 days on a coding hobby, not actually crunching some data.

Correct. I wanted to spend 26 days on my bootstrapping hobby, not actually crunching data. I did that and this app is the result.

Great idea (despite the dubious legality).

Perhaps a section on what can be done with the data? Your potential clients are probably the type of users who don't know why they'd need such a tool. And an explanation of the sentiment analysis?

That's a great idea. If my app survives suspension I will do this. Thank you for the suggestion.

> Up to 10k rows of data.

uh, how is 10k tweets (1 second of data?) enough?

For many use-cases 10k rows is more than enough. For example, most people have fewer than 10k followers and downloading a CSV of your followers is something many people want to do.

Charging for web scraping feels scummy somehow. Can't quite put my finger on why, since I wouldn't feel guilty about scraping twitter myself.

The app does not "scrape". It uses the Twitter API to access the data and packages it up in a nice format (CSV or JSON). There are literally hundreds of tools, paid and free, which do this for various platforms on the internet.

I'm not too familiar with the Twitter terms of service, but I would be surprised if selling access to their data wasn't a clear violation.

Well they're not selling access to data, the data is freely available. They're selling a service that packages that data for you.

Data which probably was obtained through the API which from my interpretation of https://developer.twitter.com/en/developer-terms/agreement-a... is not permitted.

If obtained through scraping vs the API it also isn't allowed via the regular ToS https://twitter.com/en/tos and their robots.txt

So either way sounds like a ToS violation. Which Twitter is usually pretty aggressive about shutting down.

I can’t see how this is true as there are tons of companies selling access to data obtained from the Twitter API such as social media monitoring companies (Sprinklr, Meltwater)

Those companies have business development deals in place with Twitter that authorize them to do this.

There are plenty of other non-enterprisey companies that sell Twitter data as well*

*I was a founder of one of them.

Interesting. Do you have any advice on how I can try this again without getting suspended?

If you are creating a Twitter app, and letting users login via Twitter and then using those user tokens to make API calls, that’s a good first step, I am not sure if you are doing this already.

Other than that, I would just make sure you position your service as something that benefits Twitter, describe it as something that allows you to create better tweets or engage with your followers better. The internet is filled with apps that sell Twitter data like Mention.com, SocialRank, and Social Animal.

Thank you this is really useful info. I am indeed doing as you say, and I think you're right it's the positioning that has got the app in trouble.

That's true in a sense. But I would still be paying them money to receive data from Twitter. So in a sense, yes, I am paying for Twitter data. In any case, I have a hard time seeing how this wouldn't violate the ToS.

As a counterpoint, stock data is freely available (through online websites, your broker/app, etc), yet it costs thousands of dollars per month to actually license and acquire such data through the proper channels. Trying to build a business based on scraping such data (or even using your broker's free personal-use API) will get you in trouble pretty quick by the exchanges.

You can’t package and redistribute tweets only tweet ids.

twint does this for free

It's last commit was on March 2021. Is the project abandoned?

That’s not even that long ago? What?

Idk. I just genuinely wanted to know. Because there are many open issues. There is even an issue asking whether it's abandoned.

I was thinking because it's twitter and it doesn't use an API, twitter might introduce breaking changes and Twint has to catch up often. Just like NewPipe, yt-dlp etc does for YouTube.

Twint is an amazing piece of software. It seems to be primarily for technical users who are able to use git, pip, Python, command line, and Docker. TweetFeast is for a different type of user who just wants a user interface to get the Twitter data they want.

Reply on news.ycombinator.com