Hey Twitter: Give us our Tweets

If I whisper the most beautiful poem over the telephone to my lover today, am I going to complain about the phone company not archiving that call properly when I want to revisit it 20 years from now? You go into Twitter knowing it's about the ebb and flow of current conversation, not about what was said when it first came out. If Twitter wants to eventually offer that, that's cool, but that's not the case right now, and you shouldn't expect it any more than you should expect 7-11 to keep their security footage of you for your perusal indefinitely into the future.

edit: from Twitter's Terms of Use (http://twitter.com/tos): "This license is you authorizing us to make your Tweets available to the rest of the world and to let others do the same. But what’s yours is yours – you own your content."

You own your content. You are responsible for it.

This is precisely why I built http://tweetsaver.com ... I wanted a nice way to search through ALL my old tweets AND keep them around. I'm always retweeting awesome content, favoriting interesting things, and sometimes I just want to remember what I said without having to click MORE MORE MORE MORE for 100 pages to find it on twitter.com. Maybe someday Twitter will turn on a feature that kills my app, but for now I think it's the best way to archive, organize and search the great content that flows on twitter.

Point rss2email at the rss feed of your twitter account, and then set up your mail server to filter those emails into a special twitter folder. Job done.

Archival of 'cloud' data is an issue I've been thinking a lot about lately. People are putting so much of their lives on the internet these days without generally giving too much thought to permanence and availability. Twitter's only been around a few years and people are already running into retention issues. The tweet from the article will still be valuable 30 years from now - will twitter even be around then? Will all the tweets from the current system have been migrated to whatever tools we're using in the future?

I've been playing around with a project to locally archive a bunch of data sources that interest me (email, instant messaging logs, Twitter, SMS, some blog and social news comments) in a straightforward and open data format. Unfortunately this type of tool might be something that most people don't realize they need until it's too late.

I value my past tweets too but twitter is having a hard enough time keeping the service up as it is. Id rather they focus on making the api more reliable before working on giving access to older data.

It really is a shame they haven't provided any mechanism, even if it were less convenient than the standard API interface.

I've seen a lot of talk from Twitter that "you can trust us", "we still have all of your tweets", but it really doesn't help much if there isn't any way to get at them. Has there been any discussion from Twitter that this is even on their radar as a near-term priority?

I was sort of surprised that the Library of Congress work did not include some sort of web.archive equivalent that allowed anyone access to the entire database — Twitter has shown a willingness to let other people solve hard problems it didn't want to, including when it pointed developers to a 3rd-party firehose for a while.

Well, I think you just found a way for Twitter to make some money. Remove the theoretical access and Web limit for accounts or increase limit to impossibly high number.

It is much akin to NYT charging for access to old articles.

They may have to store more tweets in the current database or charge a retrieval fee to obtain it from the archive. They could give it away for free too...

I'm going to plug Anil Dash and Gina Trapani's open source PHP project ThinkUp (http://thinkupapp.com/). It started off as a pet project by Gina Trapani, but now it's used by the White House to try to engage the American public more.

ThinkUp can archive your tweets to your server for any reason you want to. Not only does it archive tweets, but it also allows you to view some nice analytics such as a map of where replies to a tweet came from, a chart of your followers over time, who your most active friends, etc.

But wait, there's more! If you check it out right now, we'll throw Facebook integration, for absolutely free! That's right. For absolutely nothing, you can archive your Facebook statuses, and preserve all those precious memories.

You can get all this for $0.00. That's right, $0.00. Just go to http://github.com/ginatrapani/thinkup/ and download the code today. You can create plugins for it, modify it, and do anything to it the GPL allows you to do.

Yeah, this is a frequent pain point for a large number of users. If you search twitter for "how can I see my old tweets", there's thousands of people wanting that feature.

No wonder dozens of services (or even curl-based shell scripts or tutorials) are available to back up the tweets (shameless plug for my own http://sparrw.com/, which focuses on easy searching of past tweets and treating tweeted links as sort of auto-bookmarks, which is what I'm often using Twitter for).

But all this works only if you're quick enough, and set up some sort of backup system before the magic 3200 limit :(

Be careful what you wish for, Zach. One of the defining features of real-time is that it's very spur-of-the-moment. Unfortunately, that often leads to poorly thought out posts. In the current scheme of things that weakness is mitigated by the fact that sufficiently old posts are effectively inaccessible.

Can you imagine the implications of an account's entire tweet archives being open to the public? A prospective employer could look up every mention you ever made about them or their industry. Venting tweets would be forever inscribed in history. That's not good. It's bad.

Preservation of Twitter and other real-time services is definitely important, but not just for personal use. I'm also interested in viewing the tweet history of my friends, evolution of certain hash tags, etc.

My startup http://keepstream.com is involved in this real-time curation (Twitter now, more services later). I would love to chat with anyone interested in the subject; my contact information is in my profile.

There is also http://jetwick.com where you can search your (and also others' !!) tweets. If you searched the first time you probably won't get results for a user. But please ;-), come back 10 minutes later. Then 50 tweets are searchable and stay searchable a long time. What long time is depends how popular you make this service ;-) I.e. if you are using the service regularly you can create a free (!) archive. Again: no payment necessary (this is important for us!). and even no registration at the moment.

There are a lot more features read through the about page (e.g. filter via dates, languages, sort against date, query dependent trends, ...)

Check out http://pongsocket.com/tweetnest/ .. it's open source, free, PHP (easy to install), yada yada. It's a simple PHP Twitter backup system that also presents your tweets in a reasonably attractive archive on your own site. Example: http://peterc.org/twitter/

(Yes, there's also http://thinkupapp.com/ - but it's more complex and has more features/supported services. I found TweetNest crazy easy to get started with though.)

Thankfully I took a CSV dump of my tweets 2006-2008 but I'm missing quite a few in the gap :-(

The library of congress has them, if we could get a copy we could index it all somewhere to re-construct a copy of people's early timelines http://blogs.loc.gov/loc/2010/04/how-tweet-it-is-library-acq...

I was thinking recently about getting all the tweets from 2006, it should be about 2 million tweets, so should be possible to fetch by sequentially walking through the ids.

Heatmap of your foursquare checkins. I'm sure these guys would be up for doing it for geotweets.

http://www.weeplaces.com/

Incidentally, the limit of 3200 tweets via the API is the same as the 160-page limit on the website. 20 tweets per page * 159 additional pages = 3180.

i like it that tweets are ephemeral. its good that way. why the need to archive every-fucking-thing? imagine how shit the world would be if everyone could browse everything they ever said? rather than the things memorable enough to remember. or memorable enough to someone else that they'd remember? imagine how many useful people would become lawyers?

WHAT ARE YOU THINKING!!!!!!!!!!!!???????????????

You can always favorite those really important tweets, or even resort to printing them in a compendium by Tweetbook.com. Some of my earliest tweets in a book form that I can go back and reflect on.

I agree that Twitter can do a better job of allowing us to search tweets by topic, but that's a 3rd party app waiting to happen. In the meantime, take a screen shot and post on your web or in flickr.

I'm glad Twitter doesn't keep a record of my tweets. Perhaps for what you want, they could make it an opt-in service, but for invasion of my privacy, I'd just as soon not. And for their service load (media needed to store your tweets and server bandwidth time needed to make them accessible to you) I can well see how it might be a payable service.

Um. Hand on a sec... on the bottom of the same page you link to:

  Twitter still maintains a database of all 
  the tweets sent by a user. However, to 
  ensure performance of the site, this 
  artificial limit is temporarily in place

So maybe you can't do it now - but its not lost (yet).

I recommend using The Archivist, which is free and will automatically save your tweets forevermore, for easy download at any later time. I've used it a couple of times to dig out old tweets of my own! http://archivist.visitmix.com/

They could simply slap a 'last visited' flag on ever tweet and discard those that had not been looked at after a year or two. You always have the option of backing up your own data, how much storage is it going to take to periodically scrape you tweet history in to a text file?

I recall hearing that its due to scaling/db limit. Something like they cant actually go back further due to memory limits. Ive no idea if this is true or not. If the Twitter team skype in at the next uk #devnest I shall try and remember to ask.

I've heard (second-hand) that Twitter isn't discarding users' tweets after they hit 3,200, they just don't provide API access to them. They're still sitting in a DB somewhere, just inaccessible currently, until Twitter provides access to them.

I had always assumed that this would become a feature of Twitter if they ever offered a pay version of the service (higher API limits and complete access to past tweets). I imagine that some people would pay for that.

I agree, I save old emails, old letters, old receipts - it sucks that I don't have access to all my twitter messages, wtf? And please don't tell me that "you're entitled to a full refund" crap.

I've been using http://www.backupify.com to backup my twitter account. It was really interesting to go back over my first tweets.

I completely agree. I saved the link to a tweet from Biz Stone about an app i launched 2 years ago (twootball), but now it's gone.

It's just a technological failure (because it requires too much cpu power to dig through a non-indexed non-relational db) and they don't want to show their shame of poor design.

Can you imagine any other service surviving where you cannot lookup older data?

BTW Google has a social search that indexes twitter and facebook but it's not complete by any means.

as a journalist -- I loved this post. there is a lot we could mine and then synthesize. really need better ways to try to tell more _meaningful_ stories and cover all this 'now, now, now' content.

@coryhaik washington post

Didn't someone recently create a way to go back and see your first tweet?

we should be talking about how to build the federated social web as the solution, not just bitching about the problem

backupify - it's the new ronco showtime rotisserie - just set it and forget it.

It's funny to me to see people prizeing their tweets so highly.

I reviewed mine a while back and realised they were inconsequential drivel, so I deleted them along with the account. Now I have a read-only account for reading the tweets of some folk who tend to be funny to read. No techies trying to demonstrate value, no not-that-interesting acquaintances, no naive political rants from relatives - much better.