Ask HN: What is nitter and why does it still work? | Hacker News Reader

Ask HN: What is nitter and why does it still work?

54 pointsblueridge20 comments

You can't access X without signing in, then there are the rate limits if you are signed in, but these things are not an issue with nitter. I went from checking Twitter as a logged out user every few days, to not using it at all after the access changes, to casually using nitter to check in on a few accounts I like.

https://nitter.net/search

View on HN

Nitter currently uses so called "guest accounts" and a proxy network. Start here if you want more details:

https://github.com/zedeus/nitter/issues/983#issuecomment-168...

The required proxies are the reason currently only nitter.net is working, but not the other hosted instances.

other instances are working. they are denoted at https://github.com/zedeus/nitter/wiki/Instances and more readably at https://status.d420.de/

https://github.com/zedeus/nitter/

I think it uses the same functionality as the guest account feature in the Android app as of now.

If that's true, isn't it just a matter of time before it gets shut off? I don't really see a way to keep this working without somehow crowdsourcing accessing X. Even with an automated way to create fake accounts with phone verification, it would eventually be detected or guarded against.

I mean, possibly? The loginwall thing is a fairly obviously silly strategy which they may not be excited about doubling down on, though (I would expect it's fairly likely that the loginwall gets reversed at some point; they've already stepped it down so that non-logged-in users can see tweets).

But honestly, at this point, _everything_ about twitter is up in the air. Who knows what state it'll be in in a year.

That’s how archive.today works with news publication site accounts, will be fine if you have to make round robin requests with a bucket of X accounts.

I would disagree here, I think it's different for several reasons:

- The volume and velocity of data and API requests needed for X is much larger than scraping the occasional news article. You'd need a huge number of accounts to make it work.

- X has the technology to reliably detect such accounts unless they are incredibly well "hidden" by e.g. crowsourcing requests through regular accounts with normal browsing behavior

- X has the incentive to reliably detect such accounts as it has shown in the past by shutting off data access. It seems to care much more about this than most news websites

I believe the only reason things like Nitter still work is that it's not yet big enough for X to care about and invest resources into guarding against it, especially with all the other stuff going on at X right now. But it's only a matter of time before that changes.

I can tell you that a project is already doing this today for archiving with no signs of mitigations (request denials, poisoning the responses, etc). It’ll likely eventually end up in the usual places when you want to see something from way back in the day.

They arent scrapping the whole site. The entire site is 99% spam. Its scrap on request, if its not already cached.

That should be “scrape”/“scraping”, unless you mean they should be throwing the site away :)

Sure, but that doesn't make a difference. Even the top 1% of the most requested timelines and tweets are huge amounts of requests. That's many many orders of magnitude more data then new articles and requires a large number of accounts.

> But it's only a matter of time before that changes.

If anyone at the helm is paying attention, which seems iffy these days.

>- X has the technology to reliably detect such accounts unless they are incredibly well "hidden"

Has, or had? With massive layoffs, they might not have the staff or know-how left to do this, or the time to implement it.

>- X has the incentive to reliably detect such accounts as it has shown in the past by shutting off data access. It seems to care much more about this than most news websites

OK, but still wanting to do something, and actually doing it, are two different things. I'm not convinced this hollowed-out shell of a company still has such ability.

>especially with all the other stuff going on at X right now. But it's only a matter of time before that changes.

This seems to assume X is going to continue to survive and in fact thrive; I don't think that's a good assumption.

I self-host a LAN-only Nitter instance for personal use. In the last few months of Elon trying to lock out third-party clients, it's gone from automatically acquiring tokens, to tweet access breaking and coming back and eventually breaking again, then only showing "top tweets" for users rather than full post history, to having to log into Twitter in a browser and extract some cookies/headers using Developer Tools (and failing to show user post history because they tried using RSS feeds that got pulled). The latest development is the use of "guest tokens" (https://github.com/zedeus/nitter/pull/985) that you have to create and install by hand (running a Node script) because Nitter hasn't automated the process yet. This works well for viewing all tweets, but hasn't been merged to the master branch yet (but nitter.net and the AUR nitter-git package are already using it).

Previously, found using HN search: https://news.ycombinator.com/item?id=37397715

Spoiler: because apparently, it's API access somehow hasn't been pulled yet.

The author made a post bragging about how easy it was to circumvent rate limits since X doesn’t even limit by IP address, so they had 1 address making requests for like 1000 accounts without any problems.

Historically it worked by using Twitter's undocumented internal API, but I have no idea what it does these days.

Its not even getting rate limited anymore.

When I checked this a few days ago, Nitter wasn’t returning the latest data, so aggressive caching would be one of the tricks they’re using.

Nice try Elon.

No clue how it works nowadays

Reply on news.ycombinator.com