Twitter archiver: Make your own simple, public, searchable Twitter archive | Hacker News Reader

Twitter archiver: Make your own simple, public, searchable Twitter archive

github.com

167 pointsaaossa20 comments

Another shameless plug: I made a cross platform "native (Rust with Webkit)" desktop app that does the same thing (Win/Lin/Mac). It also allows you to see your "profile" afterwards in the app, search within your Tweets, and allows backing up things like mentions, responses, likes, etc:

https://github.com/terhechte/twitvault

Also, the reason I made a Desktop app is that all other tools usually require somebody to know the Terminal, or how to set up a Webserver. Twitvault can also be used by non-tech people to archive their data.

Or journalists and activists will use to keep track of everything said on twitter to use in online hate campaigns when the time is right.

Forgive me for jumping to malice, it's that I've been conditioned to assume the worst intention of activists and journalists, by activists and journalists.

Also, you can archive your Tweets with the Wayback Machine https://help.archive.org/help/how-to-archive-your-tweets-wit...

YES! Thank you, I have been looking for a way to archive my Twitter account. I've been considering deleting mine but this was THE blocker! Thanks.

Does anyone know of a means of accomplishing this without access to the account? A friend passed away last year, and I would love to archive those memories.

I hope you’ll get a better answer, but a minimal solution is to use snscrape (https://github.com/JustAnotherArchivist/snscrape) to download a raw JSON dump of a user timeline. Some caveats: 1. This doesn’t work for all users (e.g. some accounts seem to remain deindexed after unsuspensions). 2. While you can get rid of 90% of the file size by removing fluff columns (e.g. related to processing media and emojis), keep a backup. When importing such JSON, I unwittingly did a lossy data type conversion, and it can be irreparable once the tweets are deleted.

$ snscrape --progress --jsonl twitter-user jack > jack.json

[PS With courtesy rate limiting, depending on the account size, you may then script a way to send each tweet URL for archival in the Wayback Machine.]

waybachmachine is a bless these days.

Sort of a shameless plug, but you can use this PHP script to generate a lazy-loading basic webpage with all your tweets that you can host:

https://github.com/donohoe/twitter-archive

Example:

https://donohoe.dev/twitter/

Nice haha remember when you could search for something on Twitter and it actually returned what you typed in? I've tried so many times to find an old tweet of mine and had no luck. Needless to say I recently grabbed an archive of my data so I can just refer back to that, and never need to go to the sub-par Twitter website anymore.

Anyone know how to download a conversation? I want to mirror my tweets to my website with the comments.

It doesn't come in the Twitter data export. It's also impossible to get a tweets comments from the API as a normal user? (> 7 days of conversation threading only available for academic institutions and enterprise)

You need to use one of the leaked API keys for Twitter's app. That will get you the ability to call the privileged APIs.

Hmm, just tried that. Had to switch to the v1.1 API (v2 does not work with the extracted API keys, you get a `Sorry, official Twitter clients cannot use this API. Register your own.` error), but it still doesn't have access to the premium search API. You get a `Forbidden: Authentication succeeded but account is not authorized to access this resource.` error. Tried some other third-party leaked API keys as well and no luck. Standard search, even for the app keys, I'm pretty sure only gives you 7 days back (recent tweets)

Figured it out.

I found if you make a new developer account and application, you can fill out the Elevated credentials form and you will be automatically accepted (no waiting or manual process). This will get you access to the premium v1.1 API sandbox. You get 50 queries in there against the full history endpoint, which I used to grab all tweets replies to my account since the beginning.

No recursive replies but meh, better than nothing

Man, wish I did this before my account got suspended.

Is there an easy way of downloading the favorites?

v1 of the API only gives access to the last 3200 favs.

v2 of the API improves on this a good bit, but I haven't tested it thoroughly yet.

shameless plug, but try http://birdbear.app

Also wondered the same. Even more useful might be a way to download all tweets from certain lists.

The NSA should be required to give you all public information that they have on you... it would be a convenient place to get everything.

Reply on news.ycombinator.com