Introducing Twitter premium APIs
blog.twitter.com
blog.twitter.com
---
twit
noun informal
noun: twit; plural noun: twits
a silly or foolish person.
@ and hashtags came from IRC and retweets (back when it was the RT....) just evolved
edit: although disappointingly he seems to use this fact as quite a personal branding tool now
[0]: https://en.wikipedia.org/wiki/Chris_Messina_(open-source_adv...
The problem with Twitter was more serious: they never gave a concise and frictionless way to pay for the API. For example, Google innovation with AdWords was providing a self and quick service to put ads while other providers like Yahoo required a process with a lot of friction. Twitter logic was prehistoric.
Except that they didn't. There wasn't even any discussion, it was just "We're capping OAuth tokens.". Developers offered to pay for access, users offered to pay for the ability to use 3rd party apps.
If they didn't want to go down that path then Twitter could've easily required developers to show monetised tweets/ads.
They killed a community due to arrogance. And it backfired big.
I keep recommending everyone to abandon the official apps and use e.g. Tweetbot because the official apps suck compared to 3rd party (commercial messages alone is a dealbreaker) - but do peolpe then take my advice and go download tweetbot only to find they cant auth? That would be bad...
What genius thought “let’s punish developers in our ecosystem by putting a cap on their lifetime earnings” should be taken out back and summarily executed.
Previously (sometime last year, IIRC) Jack Dorsey asked developers for a second chance. I don't see how this lines up with that, is it that now we can pay them to access artificially capped functionality? This was never the problem for me, and in the meantime, nearly all of the devs seem to have [wisely] moved on.
I'm not interested in participating in another one-sided relationship with a company who behaves as Twitter has - changing the functionality and rules at the drop of a hat.
Twitter then limited this API endpoint to data from the past seven days only, and now just introduces a paid version for only last 30 days data.
I find it fair to charge for elevated API access. On the other hand, the 30 days limit for the paid version is pretty ridiculous. Makes it more worth to just scrape the free online advanced search...
"The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets."
Scraping the Twitter website is against the terms of service and may result in permanent suspension of access.
Scraping is always against the ToS, so no surprise there. I doubt it's really ever stopped anyone.
And what is Twitter gonna do about it? I never sign the TOS and scrape as much as I want.
A.k.a.: they will try their best to IP ban you.
Quoting a company that is essentially saying they had to seek funding because of twitter limiting API access is an odd choice for this announcement.
I know it's the current thing to bash on Twitter, and I'm sure there are decent use cases out there, and glad they're finding extra revenue streams outside of ads, but the speed and level of iteration seems rather low considering the resources they have.
Of course, I'm just a username on a website and have never scaled anything to 300m users, but like a lot of people, think Twitter could still be great... but seemingly still has a lot of miles to go. And highlighting that in an announcement post seems... strange
Twitter is not a technological marvel. It's something Oprah promoted, and Donald Trump bitches on.
And now, it's an API with a (social) class-based access, and without irony, also promotes itself as a leader in conversation and exchange.
(That doesn't mean that Twitter isn't too big or grew too early, but it does mean that none of those nowadays qualify as "see how you can run a big product with 30 people")
No one ever held Paypal and eBay responsible for what they did because it was so esoteric. It seems that there is a government investigation that is going to lead to a nice graph of botnets used for social manipulation which is cool, but I doubt we'll ever get a proper accounting of the really inventive fraud that dominates what twitter is.
You'll need 30 engineers just to build and maintain the datastores, caching mechanisms, and service deployment infrastructure. You then have the actual services, real time streaming platforms, streaming applications, batch infra (hadoop), analytics pipelines, notifications, security, anti-spam, recommendation/machine learning pipelines, ads infrastructure etc to name a few. Can't find it now, but there should be a circa 2011 architecture of twitter in one of the blogs somewhere. I recommend you have a look at it.
If you're comparing it with WhatsApp, note that WhatsApp messages are mostly 1->1 and it's easier to shard the servers by recipient. Also, not having to bother persisting all the messages forever simplifies the system by a lot.
[1] https://blog.twitter.com/engineering/en_us/a/2013/new-tweets...
Worth noting WhatsApp was written in Erlang.
Relevant article: https://www.wired.com/2015/09/whatsapp-serves-900-million-us...
So it seems choosing the right technology can have a massive impact on a company's trajectory and chance of success.
I do wonder what the thousands of engineers are doing. I imagine there's a lot of fire-fighting, but is it only that?
It could be the case that twitter is constantly falling over, still.
I've read bits and pieces, heard rumours and it'd just be nice to see what's going on. Unless I see something like this I'm just gonna think "BOY that's a lot of developers" whenever I remember Twitter's headcount ...
Twitter validated it's own employee count with an absurd overhead.
It has concerns of scale, performance and user experience. So does GitHub, Shopify, Pandora, Spotify, Snapchat, LinkedIn, MailChimp, WordPress.com, etc, and yet, manage will much less resources and overhead. Oh yea, and fucking Netflix - which - as far as I can tell - has much more and difficult problems to solve at a much faster pace. They employ 4,700.
As of this comment, Twitter employs 3,583 people.
What they needed to do was hire 100 talented people, allow third-party clients, try and occasionally acknowledge the developers who implement their platform, and stopped with the asinine "we're trying so hard guys" blog posts. We get it. You're having a hard time and you want everyone off of your grass. That's fine, but don't come asking for sugar at my house 'cause you ain't Beyonce.
With Twitter everyone's data is interconnected. With Netflix, you could theoretically have each server just be completed isolated from each other (modulo account management).
Netflix has operational problems to solve in terms of bandwidth, though. There's definitely major difficulties there, but they're of a different variety to the high coupling Twitter deals with
420: Enhance Your Calm
[1]: https://developer.twitter.com/en/docs/basics/rate-limiting
I feel like twitter must not be taking spambots seriously, or has malicious intent to keep them there, because out of hundreds (thousands?) of employees they can't get a team onboard to shut these obvious accounts down. I mean, it's been reported and it still exists. Either the report log is huge or it's being ignored.
What's the deal?
My own recommendation: disassemble any official app (for example, the Android one with apktool) and get the tokens and endpoints and do whatever the hell you want without paying a cent.
And you can do it if you want. I do it. I'm sure many more people do it.
By spamming I don't mean the usual "click on this link and I'll show you my tits" spam; if you create a useful bot that sends "expected" mentions (for example in response to mentions you receive, and not just spam) it will get banned in a matter of hours. With their "secret" tokens, it won't get banned.
More importantly: they introduced a lifetime cap on the number of OAuth tokens that applications could create, which meant a lifetime cap on how many different users a Twitter client could support. Once you hit your Nth user, you could never have a new person authenticate that client.
That killed off basically every decent third-party Twitter client.
Step 1 - build a product. make sure it has lots of bugs and is unstable.
Step 2 - roll out a premium product that's the same product, just stable, and less buggy.
Profit.
How is charging for an API not a finger to the developer community? Facebook for example doesn't charge for their API access. They charge for the products that the API has access to. If I have access to a product, I shouldn't need to be charged for the APIs to access that product
Compare to FB which is just a nightmare of trying to guess and figure things out. It's so bad. If it reflects their underlying model at all, their main codebase must be very difficult to work on.
Charging is a welcome improvement over the previous situation: Deal with Gnip or other data brokers. They're "Enterprise". Get no information, have to deal with a long sales cycle and sales team, have to wade through tons of BS to figure out what you need. It sucks.
My bad experiences have to do with the actual API calls themselves which aren't stable and occasionally times out or errors out.
If anything I wish they would improve their API infrastructure as a whole instead of offering the improvements only on paid accounts. I can see charging for rate limits but not for system stability.
Also, it's in solidarity with developers who invested even more than myself, years of time into apps, to be betrayed.
That's an emphatic "hell no" from me dog.
I'd still rather everyone move to Mastodon.
It took huge amounts of effort to get them to return our inquiries. Given the huge amount of money we ended up spending that seemed ridiculous.
But I guess that just got us ready for the whole "everyone must move to Gnip 2.0, but no we won't merge the multiple pull requests to make our official client work with our new API"[1] thing. Wow, that was fun...
Mastodon is coming for Twitter.
Thought it was a relic from the third party client days.
I suppose that's understandable, but I think it would be cool if they implemented something more akin to the AWS pricing model (S3, for example, as tweet streams remind me more of this than of enterprise-level pricing).
What is Twitter doing to combat this? For example, I reported this account: https://twitter.com/ksfAKBARI for being spam account about 3 weeks ago, and nothing has been done yet. I noted in my report that it has a massive follower network of hundreds of other spam accounts. Why isn't Twitter at least tackling these extremely obvious spam accounts?
Even so, automatic posting is explicitly allowed by the developer ToS, with the only restriction that you don't cause undue load (which appears to be around how often you post).
To be clear - the only things which reliably get accounts banned on Twitter are Doxing people and actual terrorist accounts.
However (based on how Gnip and the public API work), they do some level of spam filtering by default. On some APIs you can turn that filter off.
Hooooly shit, this explains so much.
There's free sandbox access, and then tiered pricing for increased access and additional features like counts and the enrichments.
Edit: The link points here:
> However, this left a gap that made it painful for growing businesses to deliver scalable solutions to customers
How about this: close your monetization gap yourself instead of pushing the problem downstream?
Twitter kept themselves in the lead - not just by delivering a superior user-experience (Nazis and trolls notwithstanding...) but also cultivating a developer community in a way that kept people off rival platforms - before stabbing them in the back in recent years (e.g. no support for polls in third-party clients).
This announcement is a continuation of the strategy: Twitter is making more data about their platform available, other platforms aren't as open, so people will stick with Twitter because their data is available - and it also benefits them directly because these "Premium" APIs do cost money to access:
> Pricing for these elevated tiers of the Search Tweets API starts at $149/month
Their search product started as a third-party integration (Summize). Their official iOS app started as a third-party app (Tweetie). They've benefited immensely.