Twitter cut off the ability to read a tweet by fetching its URL with a HTTP GET
twitter.com
twitter.com
> browser. Please enable JavaScript or switch to a supported
> browser to continue using twitter.com. You can see a list
> of supported browsers in our Help Center.
So long then, Twitter.
Consider for a minute the number of services/programs that just got screwed, the number of people on slow internet connections they just cut off, the number of people on low-end mobile devices that just lost access to local community discussions - the list goes on.
Was it really so much to ask for a basic HTML interface to display text and images? In terms of resources, surely this is quite easy to support? I've been using mbasic.facebook.com without JS for years, for example. Sure, some features are screwed, but for the most part it works.
I just append everything after the twitter.com domain name to https://nitter.net to read the tweet in question.
For example, the tweet this HN post links to could be read without javascript here:
Makes it annoying when someone links to Twitter as usually I'll have to sit there refreshing a few times before I can actually see it.
Sadly, when you submit links from nitter to HN they get automatically demoted (they won't even have a discuss link and will never see the frontpage).
I don't know the reasoning but I guess the argument could be that
- it violates original content rule (that official sources must be used).
- some engineering just went into HN to show the user-name part of twitter posts in submitted links. (I took that as a reasoning why HN prefers twitter links rather than nitter - suppose that could easily be expanded though).
- potential legal reasons (? I can't think of any but maybe HN respects the claim of twitter that the data belongs to them etc ...)
just speculation but fact is there is no nitter support on HN.
YouTube -> Invidio.us
Instagram -> Bibliogram
Google Maps -> OpenStreetMap
reddit.com -> old.reddit.com or i.reddit.com
[0]: https://addons.mozilla.org/en-GB/firefox/addon/privacy-redir...
This domain is dead, I currently am using invidious.xyz
The day Nitter is killed is likely the day I stop using Twitter. Yes, I know you can host your own, but if Twitter decides to actively fight against it the best case scenario is an annoying cat and mouse game that makes the service unreliable.
I'm connecting from France, I'm wondering if it's maybe EU-only thing related to FB's latest rollout of "we disable some stuff for EU users to be GDPR compliant"...
Yes; it allows automated mass aggregation of hard, accurate, and current information, which is not only non-ideal from profit perspective but also for maintaining community active and/or engaging
curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://google.com/bot.html)" "https://twitter.com/zarfeblong/status/1339742840142872577"
Peak SEO when users are faced with more friction than Googlebots and crawlers.
https://raw.githubusercontent.com/alandipert/ncsa-mosaic/mas...
The User-Agent header is not mandatory and was never intended to be used by tech companies for denying access or fingerprinting. It was supposed to be used, at the user's discretion, to help with interoperability problems. RFC7231 specifically refers to user-agent masquerading by the user as a useful practice. It explicitly discourages using this header as a means of supposed user identification, e.g., fingerprinting.
If you are big enough, there are separate rules for you (or no rules)
Also, google has a commercial relationship with them: https://www.convinceandconvert.com/social-media-research/twi...
They should just announce that users must use a particular user-agent header value and provide a list of approved values. If no one else compiles a list of acceptable user-agent header values for Twitter, I might have to do it.
Every user should just use the same user-agent header value. That would negate any utility of the user-agent header.
Edit: ah no, they stopped https://developers.google.com/search/docs/advanced/crawling/.... I don't think 2 DNS lookups are acceptable to block a GET request but I think it can be done out of band, i.e. isGooglebotIp function can fire away a Redis query and if nothing was found, to put the ip into the DNS verify queue. A few requests later, the bot will now get banned thanks to a new record in Redis.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
This method is already used by stackoverflow to hide the sitemap.
I was under the impression that serving other content to google would greatly punish your pagerank and even pull you off the search results completely.
Edit:
- on chrom(e|ium) based browsers: chrome://serviceworker-internals/
- on firefox: about:debugging and scroll to service workers section
2. How comes such a bug is still in production for months?
Not sure how exactly the service worker was broken (I didn't have the curiosity to check in devtools, and now it's gone), but I had this exact same issue and saw other people having it right here on HN, and this was the way to solve it too.
Now I'm trying to figure out how to do it on Mobile Safari, where this bug always hits me.
I still gave it because I'm a little guy and want to get the benefits of being tapped into my professional community, to know what's up, to be visible myself etc. Such individual decisions result in this.
We users have a hard time coordinating as we have individual incentives while Twitter is one entity and can do whatever it decides.
Well, of course to decrease friction and avoid turning people away at the entrance. You get smacked in the face with the request for your phone, once you've followed some interesting people, interacted with the site a bit and will form some attachment. Then you are more likely to enter your phone to keep using it.
There's a psychological effect of feeling more strongly about retaining a given amount of already owned value compared to gaining an equivalent amount of stuff you don't have yet.
You wouldn't pay with your phone number to get a Twitter account, but you would pay with it to keep an account you already have and started using.
Dark patterns.
I've had this exact same problem about half a dozen times, and every time I have difficulty getting in contact with a real person, and not being led through an automated system that just ends up asking me for a phone number, or getting a reply hours later about how that inbox isn't monitored.
Startup = Good Open API
Middle = Start Closing API for "Quality, Security and Privacy"
Large = Block all access to API unless you are another Big corporation...
Every platform has gone through this life cycle, it is predictable at this point
Edit: And dont for get the new excuse "bots are everywhere" the go to red herring to justify anti-user anti-consumer actions on 2020
I know (in principle) how to use curl to send an authenticated API request via HTTP. That's authenticated; you have to log in. I already have a twitter client for that use case.
There's also unauthenticated API access (read-only public tweets only, as you'd expect). (I assume this is what nitter.net is doing.) Less tracking, but you still have to request an API key, so it is under Twitter's control and they could in theory cut it off. It's still not supporting the basic web principle of "you can GET the flippin' data".
Anyhow, I'm glad this is getting some public attention. I'm a little surprised that I was the first one to make a fuss about it.
How hard would it be to write a centralized 1-to-many message pipe that replicates Twitters core functionality?
How hard would it be to run such a thing? The moral and legal needs to censor content would probably be a bigger task then the technical implementation?
American giant corporation with questionable ethics: hard no.
Their approach is somewhat understandable - stopping bots is a hard problem, phone number in theory represents a material cost and a hurdle to someone who wants to rapidly register thousands of new accounts.
But no, I don't have a burner phone. I'm not going to buy a phone and a SIM just as some kind of non-solution to tech companies having a hard-on for invading peoples privacy.
> ... centralized ...
That was not in the GP’s requirements.
Edit: plus: GP explicitly asked for software.
Every twitter clone that tries to compete without first having the approval of the major media outlets will either be publicly ignored, belittled, or outright attacked.
Please note that I explicitly said "centralized". Because decentralized solutions like Mastodon have not resulted in viable alternatives for the average user so far.
Looking at Minds, I already see a big flaw: All posts are under the /newsfeed/ namespace.
So instead of
/username/1234
/newsfeed/1234
There also is no embed functionality. For Twitter type shoutouts it is absolutely vital that they can be embedded around the web.
You proved that point
I've never used Minds and dabbled with Gab when it first came online only to find that it is as noisy and worthless a place to engage as Twitter is, possibly moreso.
fwiw, I thought the same until recently but discovered they are visible with the following format:
Really, it's just that social networks work by having a lot of people, and lots of people are on Twitter because other people are on Twitter (I think there's an audience thing as well).
Though, to your point, I think Twitter has outsized influence for the same reason Google Reader did: loud nerds and journalists were always using it. Though now the politicians using it probably count for a lot too
History is full of conspiracies.
> Really, it's just that social networks work by having a lot of people, and lots of people are on Twitter because other people are on Twitter (I think there's an audience thing as well).
The network effect is real, but it is not the only effect. Twitter and other tech giants now think that their role is to protect the world from information they deem wrong. As well, prominent people in media and politics are urging them to take on that role more expansively. Naturally, therefore, they will oppose competitors whose distinguishing feature is to have less or no censorship.
The media establishment does not look kindly upon those who seek to deconstruct it.
Do you have any examples of anything "they" "deemed wrong" which wasn't factually wrong? And not "alternative facts" bullshit, regular, proven facts.
Now, you place the burden of proof on me, assuming that Twitter is in the right by censoring information and demanding that I prove that their censorship is wrong. But why don't you place the burden of proof on Twitter? Why do you apologize for censorship?
In your other comments, you say that "Only China can stop China," and you say that the U.S. mustn't respond to cyber attacks by China, Russia, and Iran, or those nations might escalate, so we must look the other way when these nations infiltrate and attack us.
One wonders what you're after.
Facebook did not grow this way, even though their relationship with the media is nearly indistinguishable from twitter’s today.
And then in 2008 it ramped up for a multitude of reasons. A politician nobody had heard of somehow managed to mobilise a lot of support via a tech platform many hadn’t heard of, and get to a point where it looked like he had a legitimate shot at becoming US President. By early 2009 it was popular enough that Facebook updated their timeline to try and combat it, everyone was talking about their growth, and Ashton Kutcher was goading CNN into a competition to be the first account on the platform to have 1M followers (he was the first btw).
Old media was late to the game here and had to play catch up to stay relevant. I suspect that was also the tipping point though which changed journalism forever into this clickbait and eyeball driven economy. If anything Twitter inadvertently controlled old media, not the other way.
So don't worry about critical mass. Pick where you want to publish based on what is important to you.
I almost exclusively use FOSS, federated networks to publish my thoughts, the only exception being this site. I engage here pretty often because I like the quality of the discussion and the rule of only contributing if you have something constructive to contribute, but ideally this website wouldn't be a silo.
Critical mass seems important for exposure and personally I don’t think exposure is important unless you are seeking to maximize something that comes from exposure, usually it’s profit.
My comment was mainly to bring attention to what I consider the primary causal factor behind twitter’s critical mass.
It's easy to set up your presence and start small, e.g. with a web app which syncs your Mastodon and Twitter accounts. https://moa.party/
EDIT: Odd, I'd like to report a bug in that site. https://imgur.com/EsGyvnT It doesn't seem to load. Or rather, I saw it briefly flash the statistics, once, before it went back to loading. The 0.3 seconds I saw the stats looked impressive though. :)
The only thing in dev console is a warning that seems unrelated: "<ApolloProvider>.provide() is deprecated. Use the 'apolloProvider' option instead with the provider object directly."
Ah, the root cause is that the graphql fetch takes ~32sec: https://imgur.com/q769ozR So, never mind! Just a bit slow.
---
Anyway, it claims 400,000 active users in the last month. What counts as an active Mastodon user? It also claims 400 million Mastodon posts (total?) which I suppose are tweets.
Interesting... Twitter has about ~17M tweets per day, iirc. If this growth is legit – a big if – then Mastodon may be gaining more momentum than it seemed.
Actually, twitter gets 431M tweets/day: https://twitter.com/jasonbaumgartne/status/13320033501922713...
Hmm. I wonder how much exponential growth would be required for Mastodon to catch up to twitter...
I have met many people on the fediverse, and at least two friends of mine have moved thousands of miles due to relationships and opportunities that likely would not have occured if the fediverse (Mastodon, Pleroma, etc) and its variety of platforms and userbase did not congregate in this manner.
One friend even bummed a plane ride on a tiny general aviation plane by serendipitous happenstance through the fediverse to make the multi-state move out of an abusive living situation.
Unbounded growth is not always good, and many fediverse instances have limited or closed new account registrations to ensure their community doesn't become the next Voat, FreeSpeechExtremist (which I was banned from for fairly benign speech) or /r/TheDonald.
There are quite a number of people using Mastodon and it is quite nice but it does feel a bit echoey all the same. At the same time there is a lot of garbage on twitter that you really couldn’t call “active users” and keep a straight face. As usual e truth is usually somewhere in the middle ...
True, but some of the more avidly pro-Mastodon and anti-Twitter regulars take it upon themselves to be gatekeepers (some nice, some nasty) if you try the dual-platform method. Just noting this for the sake of those who haven't yet done so. Feel free to give it a go, but don't be surprised if you get smacked in the face fairly early in the experience.
https://twtxt.readthedocs.io/en/latest/ is an interesting way to do it.
RSS with comments can work. Mastodon, Pleroma, misskey, gnusocial all do it. They have the added benefit that a user cannot be censored from the network entirely, only from servers by their admins, and blocked by users.
I for one would love for there to be a decentralized system that by design makes it impossible to censor any content, and if the content would be stored in a decentralized, encrypted manner it would also be impossible to legally enforce it as it would not even easily be known where it would be stored at all.
Think of perhaps a network that retrieves its content viā an union-routing-esque mechanism where information is published that cannot feasibly be altered any more once put there whose origins and locations would be computationally infeasible to trace and as it's stored in an encrypted fashion the parts of the network that do store it do not even know what they are storing.
Not a particularly new concept, Freenet has been around for over 20 years now. Tor and I2P have more traction in this segment.
And sure enough, I already found a website that says: “Please note that the Freenet network (much like Tor) attracts pedophiles and a large amount of sites contain child pornography. Some sites jokingly add a disclaimer saying This site does not contain child pornography. click here to continue.”
So the end result is that all one finds on it is child pornography, controversial opinions, leaks of government data, and the lot.
I suppose my ideal wish was a platform such as twitter that intermixes legal and illegal content.
I didn't realize it then, but if Hunter Bidens data had been put there they would have been basically impossible to censor and it might have swayed the election.
Like, this isn't a "think of the children" argument that those communities shouldn't be allowed to exist. Of course they should. But don't be surprised when people prefer more centralized and "censored" (policed/moderated) communities. For the vast majority of people, that is a feature, not a bug--and they tend to realize that and boomerang back to Twitter etc. when they try out decentralized/anticensorship-oriented alternatives.
A rather big difference is also the relatively lower level of activity on those alternatives.
You can go back and read tech media from the time about all the engineering challenges they ran into and the regular downtimes from trying to scale the thing to a tiny fraction of its current volume.
Turns out to be hard.
However limited their technology choice might have been from a performance standpoint it did give them the adapatability and flexibility to provide the fun and entertaining product that got them off the ground in the first place.
It did inevitably lead to the growing pains we discuss but I doubt anybody expected it to get as big as it did.
WRT the original comment here ... the main limiting factor nowadays is just getting people to use it. The network effect. Mastodon is a thing, and people do use it but it’s a minnow relatively speaking.
The next big challenge I think will be some kind of decentralised twitter based on some kind of blockchain (by which I mean progressive content hashing and consensus) and I don’t mean federated, like mastodon. The mastodon name is ridiculous too .. I’m not going to go into why but if you know you know, and stupid as it is it does matter.
For a while already I had the idea of building a "diet social media" software that would display information from e.g. Twitter, LinkedIn, Facebook and other platforms without gamified addiction mechanisms. Turns out this is impossible to do as all platforms have drastically restricted the way you can consume their data.
Bubble [1] is a new kind of VPN that does just about exactly what you are looking for. On FB, Twitter, LinkedIn, Insta, etc, you can scroll through your feed and:
* No ads!
* No tracking! (except what you click, can't avoid this)
* Auto-block posts matching keywords you don't like (politics/etc)
* Auto-block posts that contain links to news websites (I use this on FB, it's great)
* Block specific users whether you're connected to them or not.
* Blocked users don't know you've blocked them, because the block is at the network-level, not in the social media service.
Bubble is open source [2], our commercial offering is currently in beta.
How does it work? It's basically a WireGuard VPN with a tightly-integrated mitmproxy. So it works on any device, but the social media blocking only works on the web (native apps are cert pinned, which breaks mitm). AMA.
Oh did I mention it works on HN too? I rarely need to block a user here, but when I do, it sure is nice to have that feature!
The claim that it's "open source" a bit disingenious after looking through your GitHub repositories. It's not like you could test Bubble locally and then decide to use your hosted service.
I think the overall use case is solid, but would be much better served with a browser extension. Also, very likely mitmproxy won't really scale that well to lots of users.
Even if you launch using our paid service, it’s the same- we don’t have access to the system we launch on your behalf.
Bubble is as private as running your own WireGuard VPN, it’s just a lot easier to set up and comes with some cool features.
You can absolutely try the open source version right now: https://git.bubblev.org/bubblev/bubble
We can’t see your traffic. We can’t login to your Bubble— but you can install an ssh key and root around all you want.
We are anti-SaaS
Are you for real?! This is basically the polar opposite of what people should be looking for, no?
The content was only ever the cheese to draw us into the advertising trap. They're not in the business of giving us free content, they're in the business of selling our attention as we gorge ourselves on "free" content.
I mean Google has invented whole protocols and formats to save small percentages of traffic.
It sucks, because it adds a new refresh cycle just to see the tweet.
- Twitter having a broken service worker (saw this a lot on Twitter discussions in last weeks)
- Safari perhaps lowering the JS execution priority in background tabs
- Twitter trying to be nice and only exec JS while their page is in foreground, and hitting a bug in Safari regarding visibilitychange callback not firing (which is fixed in Safari 14 by the way)
I want this. I want it to save battery life.
So Twitter should make sure their javascript executes correctly when it is foregrounded and Safari does start executing it.
Either way, I honestly don't care too much why it is broken, Twitter could have sent a pre-rendered page down.
Any references/links for that sort of breakage? I'm looking at using service workers for a personal project (mainly of offlining, but partly for background processing). Being throttled in a background tab is fine, but I'd like it to work fully without manual interaction (a refresh) once the tab has focus.
I expect an RSS style newsreader. I just want to see what's new, in order.
IMHO John Carmacks' old school finger file updates was the pinnacle of human achievement. Just aggregate everyone's finger updates into a swanky offline mail reader style UI. Heaven.
I don't understand why links aren't links. eg Open this in a new tab.
I don't understand why pull quotes are images.
And tellingly, I don't care enough to figure it out, find a better way.
PS- One thing missing from Twitter is the drive-by downvote. Too bad.
Just replace the twitter.com part of the tweet URL.
curl https://twitter.com/zarfeblong/status/13397428401428725771- https://twitter.com/i/api/2/timeline/conversation/<twitter status id>.json