Twitter shut off API access; users volunteering their own data for an open API
omarshehata.substack.com
omarshehata.substack.com
What is the game theory here? Twitter cooperates and OpenAI defects, and we call that a win?
No other social media site (besides Reddit) had one. Facebook kind of tried, although theirs was always a lot more limited, and got shut down when it turned out Cambridge Analytica used it for widespread election fraud[1].
Both Twitter and Reddit did basically the same thing at roughly the same time, Twitter just had the misfortune to be under the control of Elon Musk, so the move was perceived as ideological.
[1] As it later turned out, Cambridge Analytica was basically a "nothing burger", they claimed to be able to accomplish a lot while, in reality, accomplishing very little. The damage to interoperability was already done, though.
What CA did show was that Facebook's statements about protecting user privacy were fundamentally incompatible with the way their API worked, so they had to shut it down because the alternative would have been to just sort of... Let it hang in the air that it wasn't hard for a third-party to build a system to completely bypass user intent in scoping their information.
(I had the misfortune of trying to write a Facebook app about fifteen years prior, and that was my takeaway at the time also... "Do people, like, realize that their whole process for protecting scraping the social network via third-party app integration is the honor system?" Turns out people didn't).
That sentiment goes back pretty far in Facebook's history
[1] https://www.yahoo.com/news/mark-zuckerberg-branded-early-fac...
For those unaware, Mastodon's APIs are extremely open and it's very easy to scrape, to the point of providing you with a "firehose" of all public posts that an instance sees, both local and federated, with no authentication required. They also have an extreme anti-scraping culture, anybody who admits to running any kind of scraper which is not strictly opt-in, even for benign / scientific purposes, is very quickly shunned and blocked. Most instances also have a "disallow scraping via robots.txt" policy by default.
The results? I posted a canary token[1] link on a medium-sized, well-federated, well-protected instance which disallows scraping, and it got hit by some shady social media crawler in a fraction of a second. It started getting hit by many other strange crawlers later on, and it still keeps getting visits (mostly from Google now).
And I don't know what to tell them, because that's not how the internet has ever worked.
1. Experts can comment, but I think the value of multi person conversational data from forums is uniquely valuable and in short supply relative to just blogposts/news stories on the internet.
2. The absolute economic value of the entire corpus of Twitter is much more valuable than any single boatforum.com like website. So Twitter has a much more incentive to lock itself down than boatforum.com.
That was was the story spread far and wide by the Apollo app developer that was believed by the gullible and angry Reddit masses.
But Reddit actually set a reasonable API price, as evidenced by the fact that a year and a half later there are still five 3rd party apps running on reasonable subscriptions:
Infinity For Reddit
Nara For Reddit
Narwhal 2
Now for Reddit
Relay For Reddit
It doesn't have to be one company controlling who gets access, the users can decide this
In this sense we have this intractable push and pull going on with just about any community of sufficient size. We have the users who might want or should want some privacy and respect and other such benefits, and then we have the people who actually invest and build these platforms, who are incentivized to deliver other things than what the users best interest might be. Either you empower users to have more money to be able to roll their own solutions, or you try and set up a world where the incentives of capital perfectly match the needs of the individual or collective of individuals, which is probably impossible to do.
If companies making billions isn't the part that's problematic, the billions part can be left out, and the real problem can be discussed instead.
People pirate software, yet people still develop and sell it.
Yes, it's not the most profitable thing done alone. Only for those who find the nice combination or feedback loop of 'fit', demand, improvement, and expansion.
If you can make billions off the thing, you can presumably handle some GeoIP/rate limiting... or simply, not caring. Anything that falls through the cracks is categorically insignificant to your grand nature.
To justify it, if one must, consider it a trial. As your friendly neighborhood dealer would say: "The first taste is free".
I don't use Twitter often, but I'd run something like that if there were strong anonymity guarantees. Seems like a win-win for everyone.
Does anything like this exist today?
Obviously the content is very different, but the technology is basically doing what you’re talking about, minus anonymizing the data.
I think this administration finally passed a law to make PACER free, which is good.
From what I can tell, it would work fine so long as the user is actively looking at those pages.
These pipeline tools can help people migrate if they want without losing that history & network (my prediction is once people see what's possible with an open API, that'll further motivate user migration, or for twitter to open up again)
As someone with a multitude of cat photo threads, 100x this.
While I'm around on Twitter, Bluesky and Mastodon, the "post history" aspect is my most pressing issue with Mastodon. Like, when you move your account, you cannot take your old posts to the new server - you're straight out of luck if your instance decides to call it quits for whatever reason. And even if the instance doesn't give up, being unable to view old(er) photos is something I very commonly notice on a bunch of Mastodon servers.
I heard that it's original use-case was that back 2008 there were no good ways to do group messaging over SMS on a cell phone. That's a problem and solution that I can understand.
But as a broader social network? I don't get it.
Things that bother me to the point that they are deal breakers:
- The character limits (there is nothing worse in life than reading through a x/N self-reply to read something long ... I'd rather file a tax return)
- Showing me posts from pages I don't follow in the feed (in 99% of the cases I'm aware of that page/profile already and have chosen not to follow it because I don't like them)
- Ads in the news feed
Some have said that they like Twitter for getting news. I have way better options for that.
I know that I'm not representative of the typical person, generally speaking, but Twitter is one of those things where I really can't understand why anyone likes it and uses it, let alone why it is so popular. And it's not that I'm hating on something I don't know anything about ... I've honestly tried to use it and get value out of it, but I've never found anything of value on offer.
I think it's a carryover of general nostalgia for how the Internet "used to be", i.e. before every site with a few million users decided to start experimenting with algorithms to max out user engagement and ad impressions, leading to the hell we see everywhere, from IG to Tiktok.
Yes, there was a time when Twitter wasn't a toxic mess. It's the last social network that was popular before smartphones took off.
Twitter, in my mind (and maybe this is perception and not reality), ushered in infinite scroll and short bites of information. Twitter is to forums what TikTok is to documentaries. I see Twitter and the "mobile revolution" going hand in hand (something that left me behind because I still dislike using a smart phone, generally, and rarely do compared to most other people).
But I guess if you're a great deal younger than me, and you grew up with an Internet where Twitter just always existed, then it might represent some earlier version of the Internet that is drastically different from what you consider to be "contemporary" (though, putting the TikTok mention aside, I'm still not sure what that view of the contemporary Internet is if Twitter is what we're comparing it to).
I guess I'm just so old that I still see Twitter as a relatively new phenomenon. Very different from the nostalgia that I feel for what the world wide web used to be when I was young.
Then you have the regurgitated messages in replies (AI or otherwise) endlessly copied and distorted. Recently it's been 5+ pages until I see a real reply, while I have 3000+ accounts blocked by now. It's getting harder and harder to find the real person who wrote that witty reply, and not the endless blue check accounts who copy-pasted it to farm engagement on their visibility-boosted tweets.
Genuinely getting more and more unusable.
Mastodon has a lot of great people, and a lot less of all this stuff.
1. I just couldn't vibe with the culture there. From my POV, Mastodon is made out of pearl-clutchers and politics.
2. So much drama. The FediSearch drama. The Raspberry Pi incident. It's just so tiring and you feel like you need to walk on eggshells all the time.
3. So much drama. You would just pray that your admin didn't get into a spat with another admin and get defederated. You could get a solo server, but that costs money and you might get blocked by a large server's admin anyway.
4. So much drama. Pray that your server doesn't shut down, because you can't import your posts elsewhere. Solo, yes, but costs money.
At least with Twitter, the rules are sort of well known, and you can follow anyone there unless they block you personally.
I heard BlueSky is good, though. Haven't tried it yet. Nostr was also another one, to get around the admin drama issue, but it doesn't seem very popular.
I would have thought a well-designed decentralized system would allow you to, for instance, download/export all your posts, and then import them all to a new server, in case you have to move servers for some reason.
The mainstream ones are centralized, of course: FB, Xitter, etc. You just connect to their web server and do everything there. Any centralized alternative would have be a well-financed corporation able to set up a huge IT infrastructure to accomplish the same task. But the whole idea of Mastodon is to not be a big, for-profit corporation, and to be in the hands of the users instead, which I don't think is possible with a centralized service.
So the only alternative is a decentralized service. But this means you have to have multiple servers, leading to the whole "federated" approach, so users can run their own servers and control things themselves, or use accounts on servers of their choice. This of course leads to all the consequences you named, and more: servers going down, servers defederating other servers, etc.
But again, what's the alternative? If the requirement is "not run by a single big for-profit company", I don't see one. If you think that having a decentralized service actually be competitive with a centralized one like Xitter simply isn't feasible, that's fair.
My prior complaint still stands though: if you want users to have freedom (of which server to put their account on, etc.), then why would you not have a way of migrating their account/data to another server, and force them to lose everything if their server disappears?
Both AI believers and doomers insist generative AI could surpass humans by every single metric that matters in social media. All we've got is first Nigerian, then Indian, and now increasingly Pakistani spammers desperately reply bombing trending posts made by 10 years alpha users, as if whoever behind it completely failed to do it and has been wasting gullible human spammer candidates faster than Aperture Science.
The 'they' being the corporations that hired thousands of people from the very countries you're associating with spammers , to review, correct and train AI models?
I probably spend more time on Twitter now than I ever did before.
There is tremendous value in having access to large quantities of data demonstrating the various failure modes of relatively intelligent minds. Nothing is being (publicly) done with this data so far, but this will not last forever.
This is something that has always bothered me about the /r/SSC community (and others): the "no culture war topics" constraint + moderators locking threads when the Rationalists start collectively behaving irrationally.
My For You feed kept turning into videos of public fights and Karen's calling the cops, etc. no matter what feedback I gave or what I clicked on.
Deleted it and haven't looked back. Won't click on a link from x.com either.
There's more than one society on Earth, so it's hard to say that "society" "knows" that smoking is bad without specifying the society. As an American who lives outside the US, one striking place where the US clearly outperforms peer nations, in my daily experience, is with smoking: Americans just don't smoke any more unless they're poor and uneducated. There's no smoking inside (except bars), there's no smoking within XX feet of business entrances, etc. In the US, smokers have really become pariahs since the 80s and early 90s. Some of that has been undone with the rise of "vaping", but still, if you hang out with college-educated people, you probably won't meet any smokers. It's just not like that in other countries: people of all social classes still smoke. Here in Japan, it's quite common to see some salaryman riding a bicycle while puffing on a cigarette, and many restaurants have smoking areas (usually sealed off with separate ventilation in modern buildings). There's even some old locals-only eateries that have (non-separate) smoking inside. Europe isn't any better.
As for obesity, only truly delusional Americans try to claim that "fat is healthy". All the rest know it isn't, even if they're obese, but the society makes it very hard to not be overweight, mainly because everyone drives cars and doesn't walk.
And nothing has changed after Twitter went private under Musk, except that the reasons for stuffing unwanted content into users’ timelines are slightly different.
HN is probably the last true aggregator site left that I know of.
It's partly why the web is so much worse. There's really no reason to create content outside of a walled platform is it is getting increasingly difficult to find an audience for it. Even blogging is an uphill battle since more and more social media sites penalize link sharing (they want you to create the content on their platform and leave it there).
That's why it's not surprising that these APIs are disappearing since the fundamental model has changed.
Or what if I could ask: "Given Omer Shehata's Twitter history, formulate a phishing scam that he would be likely vulnerable to".
The problem I see with here is that there are far more bad actor use cases for identifiable user data than good. In my opinion the main reason most social networks have stopped doing public by default and now do private by default is because not doing so opens them up to Cambridge Analytica type scandals where people don't realise what they're signing up for.
Personally if you do this, I would be very clear with your users that by submitting their data it will be made available publicly in an identifiable form. And that even if they revoke their data from your service it's possible for their data will continue to be archived by others, possibly for malicious reasons.
Cognitive security vulnerabilities like this are the thing I'm most concerned about. I think it's right to be very upfront about risks like these, and I'm even considering if we want to walk back the fully public thing and make it private / invite-only instead.
But on the other hand they'd both already become cesspools by that time, and I was still visiting them daily. And now I've quit them both which is a good thing.
Now the Web is expensive for some reason, and users have become so dependent on a small number of sites as a communications medium that the megacorporations running them feel like they have infinite leverage, and the social contract of user-generated content is completely forgotten.
they've got data from 800 users so far, with watch data on 55 million videos
The only thing we can is motivate more people to use open platforms like NOSTR where API or data/identity handling is completely different.
I'm somewhat surprised that this space feels relatively dormant compared to the more complex stuff out there.
(APIs suck)
Same journey reddit is making, starting after it prepared to go public.
- https://x.com/contextdogs (out of context dogs)
- https://x.com/_B___S (B&S)
- https://x.com/buitengebieden (Buitengebieden)
No, they haven't. They're still actively posting on Twitter.
It's just that if you are viewing a profile logged out you get the most popular tweets instead of the most recent ones.
The "independently" part is hard, could require some kind of web of trust.
See also: Simpsons, "elected to lead, not read"
yikes.
Cheapest is to stay away from anything related to the big clouds, especially Amazon with their "premium bandwidth" charges.
I'd probably spin up a storage box from Hetzner (or similar service for dedicated servers) as a first step, as you get unmetered connection. It should last you a pretty long while as most content will be fetched from caches.
Many companies offer "object storage" which is compatible with the S3 API
And there are some open source projects such as Minio which offer the same (and I forget what are the others?)