A new and improved Twitter API | Hacker News Reader

A new and improved Twitter API

blog.twitter.com

208 pointsmanuw110 comments

This is what they cite as the key benefits:

"A cleaner API that's easier to use, with new developer features like the ability to specify which fields get returned, or retrieve more Tweets from a conversation within the same response

Some of the most requested features that were missing from the API, including conversation threading, poll results in Tweets, pinned Tweets on profiles, spam filtering, and a more powerful stream filtering and search query language "

I really doubt this is what developers have been yearning for in a revamped Twitter API. Previous APIs provided plenty of functionality, it's just that too many of us were burned by the ever-shifting policies around API use. I personally went from envisioning Twitter as the social data pipeline of the web to vowing never to touch it again within the span of one year. I can't believe this announcement wouldn't touch on policy and developer relations at all.

> conversation threading

Actually, this was a major point of API failure when I last tried to use it. Did you know that, with the current API, there's no way to retrieve the replies to a tweet? This makes it impossible to build tools that, say, allow you to visualize a tweet thread, because you simply can't fetch the thread.

So I'll say that, as a casual, social, and informational user of Twitter (i.e. not business, not adtech, not academia), this announcement gives me a little hope that we might see a resurgence in 3rd party apps that provide more powerful ways of interfacing with Twitter. At least, that's my hope, maybe I'm naive that they'd really allow this.

> This makes it impossible to build tools that, say, allow you to visualize a tweet thread

it is possible -- it unfortunately happens to be part of my job description to do so

one of the sibling comments basically has it right. you can build it from the bottom up (where "bottom" is the deepest tweet in the thread you know about, one way or another).

let me tell you though, i am _very_ excited for this new API, because there is some absolutely horrible code i will be able to delete as a result of it

Is your work open source/viewable anywhere? I've written a few (probably ToS violating) plugins for Twitter myself, and would be curious to see.

unfortunately not, but luckily with this new api it shouldn't matter for too much longer :-)

Conversation threading is huge. I've had to use a few tools that keep getting broke to try to manually capture replies and threads to tweets for academic use. (https://www.exquisitetweets.com/ and https://treeverse.app/ which seems broken now)If a tool comes out that can do this easily (I'm no programmer so command line stuff if a little beyond me) then I will be very happy.

> there's no way to retrieve the replies to a tweet

My crusty old version of Tweetbot has been doing that for years...? Maybe not perfectly for tweets with hundreds of replies, but well enough.

I'm not sure which hack Tweetbot is using to do this (I think search as noted below), but you can tell it's a hack because it only works for recent-ish tweets. If you go to a tweet a couple weeks old and view details, you can see that it won't fill in the replies.

Can confirm, Tweetbot never has "all" the replies.

The way clients usually do this is by fetching all mentions to the user after a given timestamp (of the original tweet), and collecting only those that are replies to the tweet. Since there's a limit on how many mentions you can get, you might not get all the replies.

It's been like that at least since 2013.

Any idea how it's done in the API? I know you can retrieve a tweet's parent, but not its children.

You have to use the search feature to look for tweets and then look at the parent tweet ID. This is what bots like threadreaderapp, etc do. You can construct it from the bottom up if you know the ID of the last tweet in the thread of responses too, but you wouldn't know that a priori unless the user gives it to your software somehow.

But... that's not a tree -- there's still no way to discover all leaves. That's just a thread/singly linked list.

Yes, I forgot to mention that. You need to know the IDs of all the leaf nodes of the thread or else you can only partially reconstruct it.

so the api is literally backwards

Step 0: build a tree including all tweets,

...

;)

Possibly using the HTML API.

The final frontier

Speak for yourself. Their current API makes third party clients very difficult because they lack features and I hoped they'll announce something like this.

Don't get me wrong, I wouldn't mind them touching on the developers relations they want to build, but no company of their size will admit to wrongdoing in the past.

Also words aren't worth anything unless they come in a legally binding contract that can't change on a whim.

Given the examples of promoted API usage, it doesn't seem to have any backing away from their stance that they don't want third party clients. Add to that their API pricing was as I understand it not feasible for any new client app used by end users which gets any serious amount of usage and needs to move to paid tiers.

I wouldn't be surprised if you get an API that'd be better for third-party clients at the same time as all requests for API access for third-party clients are denied.

Most importantly none of these items, except maybe threads require a new API. Poll results would just be added to responses, pinned tweets would be a new flag to the search endpoint and a more powerful filtering and search language can just replace the search API. It seems weird that they are tearing it all down and forcing everyone to migrate.

Just out of curiosity: What type of application would have you interace with twitter at all? For ad purposes, some form of analysis? News aggregation?

Product support ticket systems use twitter and other social media sites to respond to customer feedback, as well as track and escalate issues. When you tweet complaints about your service at Comcast, there's probably a ticket created that someone from their support staff picks up. I think there's a perception that the quicker responses that these tools facilitate can help prevent negative feedback from going viral and doing broader harm to the company.

Right. And so, for those CRMs it is vital to not just catch the first tweet. Being able to catch a thread is important to catch the rant on the third tweet in a thread that explains the exact problem.

Any Twitter client?

if they reintroduce user streams I'll be pleased, if they ease off the per-app user limit I'll be elated. it doesn't really mean anything if they promise such and such policies or initiatives otherwise we'd all be holding our breath for bluesky. what's important is what they actually end up doing, which I guess we'll find out next week

Same here, I just don't trust them enough to build on their API anymore.

No word on the most important thing: the limits.

Current limits that keep becoming stricter and stricter make the API unusable.

I was looking into doing something for my own use, and you can't even retrieve your own list of followers or blocked users in one go! You have to paginate it, each page is counted against the rate limit!

Everything non-streaming from Twitter has always been paginated, no? Same with all YouTube, probably FB too right?

Same with everything ever, some users have hundreds of thousands of followers, I'm not sure how you would do it without pagination.

The limits are tight. It's true. But pagination is not the enemy.

I think better scaling of the pricing tiers could help but Twitter probably doesn't care enough about small devs.

That's what killed off Falcon Pro. The author released a version of the APP with a secret way of putting in your own API keys and I used that for a bit. Then he released a new version, but couldn't just grant the old people access because it would quickly reach the rate limit.

It was about that time I just totally abandoned Twitter.

Use Mastodon/Pleroma/Misskey. Fuck lock-ins. ActivyPub forever!

I don't understand what is the issue if user can add their own keys - just release the app with default set of keys or even without and let the user set their own.

Other than that, indeed Mastodon and co. are great. :)

Limits seem to be in the documentation on per resource-base.

https://developer.twitter.com/en/docs/api-reference-index

Or is this the old documentation?

Limit is your wallet

No joke. I maintain a bot that interfaces with the Twitter API to take in tweets and post tweets from specific users in chat channels.

I read this blog post to determine a) what I need to do to port to the new API and b) how much this privilege of using Twitter will cost me, and it is mum on both of those aspects.

Absolutely brilliant, Twitter.

I guess it's a case of "If you've got to ask, you can't afford it"

An interesting side effect of making your product ad-driven is what it does to the incentive structure of treating your product as a platform.

Twitter is just one example of many of companies who have created an API, asked developers to build on it, and then later burned the whole thing down.

Fundamentally, if you let developers build products that access your data in ways that users want, the very first they're going to do is remove the fucking ads. And, obviously, if that's how your company makes money, you can't let that happen. Given the choice between killing your ads or killing your API, you're going to kill the API every time.

Imagine running a casino and offering a "slot machine API". If you let the developers discard all of the losing spins, you're gonna have a bad time. This is essentially what an ad-driven company does when they have an API that lets developers separate the data users want (actual data) from the data they don't (ads).

Unless this is the idea. Never have worked with it before but my understanding (confirmed by many of the comments here) is that the current API is pretty difficult to work with. Rate limit the new API and allow less restrictive access through paid tiers.

Suddenly Twitter is no longer reliant solely on ad revenue and all of the stigmas associated with it in the current changing environment.

With this, they are no longer an ad company but a platform ala Google Maps/Stripe. Plenty of companies would be more than willing to pay for unrestricted access to Twitter for data analysis.

Ad revenue may take a hit but won’t disappear overnight

I'd love access to the Twitter API for research purposes for my writing, but I'm priced out for that purpose.

I can't prove it, but it feels like they only want big businesses researching their markets, and not people like me asking questions about Twitter itself - even though letting people do that would result in negligible additional traffic.

This might be controversial and I know people will say that they're a private company and can do what they want, but Twitter has positioned itself as society's de-facto official communication channel and I think the data on it should be a matter of public record at this point.

edit: Does anyone have experience with scraping tools and Twitter? Presumably it's hard to do.

I've been collecting Twitter scrapers recently with the idea that I might eventually do something with them. This one [1] seems most promising but I haven't actually tried it.

[1] https://github.com/bpb27/twitter_scraping

This is just my own vague experience, but I don't think this specific to Twitter, nor does it really have to do with pricing people out.

I think that public APIs in general are a thing of the past, mostly because of bad actors, security problems and scale. Look at things like Clearview AI, just slurping up every photo they can get their hands on. All it takes is one or two abusers and then everyone has to be limited.

The simple instagram API that I used to just get my own most recent post 3-4 times per day, now requires me to submit an application, with multiple size app icons, justification for my use, privacy policy, ToS, etc. All to I can fetch my most recent post from my own profile.

It's just ridiculous how every large website is clamping down on their APIs, but with abuse left and right, they don't really have many options.

Have you tried contacting Twitter about this? They've given free access to researchers in the past. They'll likely ignore you or say no, and if they say yes it may be more restrictive than you like, but it's worth a shot.

This is quite new: https://github.com/lschmelzeisen/nasty/

Doesn't seem to support "from:{screen_id}" in search, a bummer.

I ran a Twitter app for 9 years that they crippled, crippled, then eventually killed with policies. It went from “@ev loves your app,” to “your app is against our terms of service” in that time. My Twitter days are done.

Have you written about your experience? I'd just be curious to know what you built, and the series of the events that unfolded.

These folks from Twitter might suffer from a major disconnection from reality, they burned several bridges of trust over the years and now they just write a small disclaimer about them discovering the hack from this week and the new API apparently does not address issues people had with it in the past (that were used to choke 3rd-party apps). Kudos to the developers working on it directly though, I suppose it takes a lot of effort and the managers responsibilities are not yours to take home but they need to actually regain trust before anything else, no matter how fancy the API is. Some kind of "LTS" or long term contract for the API versions would be a start.

Exactly. What's their goal with the new API? Will they continue to be hostile to Twitter app developers?

Years ago, they made a distinct choice to become a business/celebrity platform, and actively moved away from the utility it could have been. That move cemented my view of Twitter as a dead-end for technology.

Twitter is not as profitable as they want to be, so the corporate wants user to do something monetizable, but users prefer what can’t generate revenue and don’t favor whatever that do, but Twitter is a private company so they have to do it eve if it would only result in its irrelevance, and that will lead to even less profit, then

This disclaimer is hilarious, it's like some terrible catastrophe happened far far away from them. But in fact it's been days of deafening silence out of Twitter, and they are fricking ground zero!

Like to think I'm quite positive when it comes to companies putting things out there, but still can't bring myself to trust Twitter's actions when it comes to their API.

I built and sold (albeit for a very tiny sum of money) one of those text-extender mini-blog tools back in the day off the back of the API. It felt easy, clean and part of the ecosystem.

Every action they took degraded that trust. I know they're a business and the data (and primarily their advertising serving and monitoring tech) is their lifeblood, but... fool me once...

One amazing use of the twitter API is to be able to mass delete your favs, rts, and tweets. It's the only thing I use it for as the API and platform as a whole is otherwise hostile.

https://github.com/MikeMcQuaid/TwitterDelete

Last time I checked (2 months ago, I think), the API couldn't fetch more than the 3600 latest liked / posted tweets. Has this changed?

Yes, the API can only get the most recent tweets but the script above can be used with the tweets.csv file provided within your downloadable Twitter Archive. You can delete tweets of any age via the API.

I had to use the archive initially to delete most of the old stuff but now I never have more than a few tweets to delete.

If you delete the first 3600 tweets, the next 3600 should now be the latest posted.

You're technically right, but the Twitter API won't return the next 3600, even after deleting the first ones. The response will be empty, even if you know for sure there's more.

Meaning the API is leaking the number of deleted tweets?

Could you make a tool to find out how many tweets someone has deleted?

No, my understanding is that Twitter has a "bucket" of the latest 3600 tweets you've posted / liked. This bucket is faster than their database to query, so once you empty the bucket, the API returns no result, and the bucket is never "refilled" from the database.

Those are not technical terms, sorry, but this is how I currently understand it so it might help you too?

Rather than “buckets”, I bet they just have a flag on each tweet in the database for whether it’s deleted. So if you delete the most recent 3600, next time they do a query to see recent tweets, it retrieves 3600, but they’re all deleted, so it filters them all out. Buckets are a little less likely implementation wise because you’d either sometimes see that the most recent bucket had very few tweets in it, or they’d have to be constantly shuffling tweets between buckets.

A friend of mine did a blog post on deleting favs recently:

https://tom.eastman.nz/2020/06/what-it-took-to-delete-my-lik...

Shameless plug, Twitter Archive Eraser was designed for mass tweet deletion. As of today it has deleted 3.4 billion tweets https://martani.github.io/Twitter-Archive-Eraser

The final straw for me was when twitter deprecated the years-old auth keys I had used without any issues because I wouldn't register my phone number.

If you look at twitter's checkered history, I don't see why anyone would put the effort into a new API without strict guarantees of future usability.

Strict guarantees of usability don't exist. Look how many technologies MS has abandoned.

Spoiler: It doesn't mention the word "client" at all. This is not going to bring back the features they took away in 2018.

> given the security incident we discovered yesterday

We discovered. Heh.

Nobody wants a new API. The new features don’t make up for all the work needed to rewrite every call. Again. Nobody wants this.

It is probably true nobody who currently codes against Twitter APIs wants new APIs. I've rewritten them before and won't be doing it again. Maybe someone new will come along to replace us. Twitter doesn't care unless it makes them more views.

Doesn't matter anyway because they already got rid of the useful APIs, which broke everything anyone I knew had written. Handwriting has been on the wall.

Sort of like boycotting Facebook, it's only takes the step of stopping to set you free.

As a user I would like for Twitterrific for example to support polls and pinned tweets. So at least one user of a third partly client wants this, I bet that there are more.

Agree. This seems like a poor attempt to marketing themselves "better".

They are pushing stuff on the public API that has been available for years to “managed accounts” via Gnip.com.

This makes the paid APIs more widely available - it’s now possible to start using paid APIs without talking to a sales person, with lower budget (but not lower prices - some endpoints cost nearly $1 per call).

They also plan to remove free endpoints 6 months to 1 year after a paid equivalent is released.

Bye bye per-user quota.

Here's there actual roadmap: https://trello.com/b/myf7rKwV/twitter-developer-platform-roa...

I'd care more if I could actually appeal my dev account being perma banned for real reason other than I failed to clearly express what I wanted to do with a app

> You cannot create additional apps because your developer account was rejected or suspended.

We got the same on one account because their emails went to spam. It's bloody weird to lock people out permanently for not seeing an email.

I got banned cause they didn't agree with what I wanted to go and I couldn't email/message/discuss with anyone why my use was not approved and allowed. I couldn't resubmit with changes to the plan and I had a proper business plan and it all went in the bin as I couldn't get API access. Shouldn't of created it all on my personal account I guess but it was an important lesson on not relying on external services

"The number of active real users and followers on Twitter has been greatly exaggerated..." - Inactive Twitter Users.

Bots and verified users have ruined the platform.

Personally, I don't have an issue with bots on a platform, as long as they are there to be helpful. In the case of Twitter, I would restrict bots from being able to reply to tweets^, retweet, or like. Only output original things (like automated messaging) or restrict it to DMs. And possibly restrict the use of @s and #s in those tweets so they can't affect trending. That way you can seriously cut down the signal to noise ratio and also make it much easier to spot a bot account.

And please put a BOT label on every bot account and tweet they send. This should be a hard requirement.

^ However, this would interfere with bots like the threading or videothis types. So maybe not apply that restriction.

Nothing to see here. This isn't an announcement, it's a pre-announcement. Empty PR, "We actually love developers (despite our sordid history), yadda yadda". Almost all of the details are TBD.

Did anyone look at the prices?? Holy shit. Are they selling gold? Those prices are ridiculous.

Where do you find the prices?

When you login with the Developer account it gives you an option to upgrade and there you can see the prices. But I'll save you that: 2500 requests against their Full Archive costs 1899 USD. That's 75 cents per request.

That sounds absurd to me. 2500 requests is nothing and what if you're testing? The sandbox level is 50 requests for free.

Maybe I don't understand well the use cases but all the things that come to my mind for these APIs would required more than 2500 requests to do something interesting. They are pricing their data like if it was some sort of unique canonical truth vault when in reality 95% of it, it's just a bunch of shitposts.

Are you getting the prices from this [0] page? I think that's the pricing for the old API, because it still hast the standard/premium/enterprise split.

> In the past, the Twitter API was separated into three different platforms and experiences: standard (free), premium (self-serve paid), and enterprise (custom paid). As a developer's needs expanded, it required tedious migration to each API. In the future, all developers — from academic researchers to makers to businesses — will have options to get elevated access and grow on the same API. > (https://blog.twitter.com/developer/en_us/topics/tools/2020/i...)

[0] https://developer.twitter.com/en/pricing

> Eventually, the new API will fully replace the v1.1 standard, premium, and enterprise APIs.

Uh oh. Using nitter's [1] RSS feeds is one of my main ways to access news. I have the feeling this is about to end.

[1] https://nitter.net/

nitter basically gets data from twitter the same way the the web client does it, using the unofficial (and undocumented) APIs

Good to know, thanks for mentioning it.

I guess they're quite used to breaking changes in twitter's API, then :) (presuming that a private API must change quite often)

I wonder if anybody tried to document that.

I hope they allow for blocking more than one user at a time at a minimum. Between the rate limit and only blocking one at a time, it takes forever to block a white supremacist and all their scumbag followers via the API. Much faster to just change your country to Germany.

Knowing Twitter I imagine "improved" means more closed and with stricter rate limits.

For the free tier, the user endpoint users/lookup which now is just /users seems to maintain the same rate limit, and looking at the other endpoints I haven't seen something stand out as strongly reduced.

I notice that there doesn't seems to be a distinction in the new API between user context and app context as far as rate limiting is concerned, so maybe that would make a difference.

But a lot of endpoints are still missing so everything is still to see!

I always read comments about how Twitter is not profitable. Any ideas as to how much of their revenue comes from ads vs. businesses paying for API usage?

Twitter only has 8 consecutive quarters of profitability. At some point hn will catch on.

I get it, but anything with the phrase "new and improved" always strikes a nerve with me. In all honesty I hope other companies learn from this, and put in the effort to bolster their defenses-in-depth, especially at a CSE-level. I have witnessed many a times, how much power they are given, and how easily it could be exploited by just a bit of coercion.

Notice the page never seems to mention 3rd party clients, the only thing many of us care about.

Keeping my fingers crossed for removal of OAuth and other complications accessing the API. I hate using libraries for this, all I want is a simple token-based REST API, something like Facebook and many others have had for years.

OAuth is how you get the token. Facebook uses OAuth just like Twitter does.

Things I would like: * Date when you followed someone * Bookmarks api.

Doesn't seems to be it.

Lots of (seemingly justified) complaints here. Is anybody on Mastadon? I have joined, but I have no followers and don't know who to follow. I'd love some connections over there.

I'm @PintSizePorcupine@mstdn.social

All these years and they still don’t have a serious API offering.

Will they also switch from oauth1 to oauth2 with this?

(they already offer oauth2 for server to server, but as far as I know it's still on oauth1 for user authentication)

Can something actually be both ‘new’ and ‘improved’?

Also, why does it sound like you’re trying to sell me a car?

Still no word on if they plan to make twitter bookmarks a part of the API. Disappointing

It doesn't answer the most important question: is it OAuth2 now?

Public roadmap is always an interesting decision to make.

Twitter absolutely deserves the bad reputation they have here, and this does nothing to change it.

Deleted comment

Maybe so, but please don't post unsubstantive comments here.

Yeah sorry for that, can't delete it myself (guessing because of the childs)

You can hide/delete it if you want.

Thanks.

Ok, I've deleted it.

Why should anyone continue to use Twitter if it is unable to guarantee someone won't hijack one or more of its accounts?

Let's face it, those aren't our accounts or our data. They are Twitter's.

All of that is controlled by Twitter, and intended to serve their business interests, or whoever happens to be controlling Twitter at that moment.

It was just a few days ago someone else was controlling Twitter.

Reply on news.ycombinator.com