The Library of Congress Acquires Entire Twitter Archive | Hacker News Reader

The Library of Congress Acquires Entire Twitter Archive

facebook.com

44 pointsfogus56 comments

I wonder if the LoC keeps dumps from Wikipedia (in relation to the other frontpage story). I would think that the full 5.6TB Wikipedia dump would probably contain more useful information on average than 20TB of tweets.

[ref: http://news.ycombinator.com/item?id=1265138]

This headline reads like one for an Onion news story

I thought for sure this was a late april fools post.

Yeah, I thought the same thing... then the fear that my tweets to trending hashtags like #threewordsforyou or #worstlies are in the Library of Congress for perpetuity.

It's not helped that the announcement about acquiring the Twitter archive is done on facebook.com.

Hopefully they'll convert the short urls to long and make some arrangements to download pictures from twitphoto and their ilk.

If not, this would be a cool project for a third-party distributed project like SETI@home or folding@home.

It's a lot of data, but not THAT much data. Nothing that a small amazon cluster Hadooping it couldn't do in just an hour or so. If anything, the bottleneck would probably be the URL shorting services themselves.

Yeah. I have a multithreaded perl script that can basically peg the database on the far end. This usually happens at about 20-30 threads. It sort of leaks memory, to the point that restarting it every day or so is advisable.

http://github.com/chronomex/urlteam if you're interested in my code.

I'm a member of ArchiveTeam (http://www.archiveteam.org/). One of our current projects is to archive all the URL shorteners we can find. We're going to make the full dataset available for public download.

This is fantastic news! Please announce the public dataset on HN when it's ready for public download.

I totally hadn't thought about that, but I will. :)

Can we report shorteners? For example, I host a private-use one at http://ajh.us that's powered by my project http://lessnmore.net

While they may well have a way for you to submit archives, I think the greater question to ask is "why?"

URL shorteners are bad enough on their own, but arguably a necessary evil -- encouraging everyone to set up their own is in no way a good thing.

I can't think of a worse service for an average user (or anyone without a datacenter, or better yet multiple datacenters) to run. By all means run your own email server -- if it craps out, at least you'll just lose your email. But if you run a URL shortening service and send out or post a lot of URLs using it, and then you go bankrupt, die, or just get bored and decide to stop running it ... now everyone who happens to have one of the links is S.O.L.

It strikes me as the kind of service that's relatively easy to run, but unless someone is prepared to run it forever, they probably shouldn't.

I am using it primarily to shorten links back to http://alanhogan.com, and since I run both domains, I intend to keep both of them running as long as possible -- and when one goes down, the other becomes pretty much useless.

But yes, I agree that it is important to intend to run these services forever.

You can add them to the master list at http://archiveteam.org/index.php?title=TinyURL . If you know it, include the charset and algorithm that the short URLs are generated from.

If you don't mind putting a CSV copy of your database up for public download, that'd be even more awesome! (http://ur1.ca/ does this, for example.) Plus then I don't load up your database server with my multithreaded, rather antisocial scraper of doom.

I work for the LoC. I think I'd of heard of this.

Edit: It is my considered opinion that a lot of what the LoC does is excessively ambitious. The project I'm currently working in is called the Born Digital project, which is an attempt to allow major broadcasters like PBS and CNN to submit broadcast material to the Library in real-time. The tools to do this, do not currently exist.

What's wrong with ambitious?

Also - I'd thought real-time capture of video sources was something the DVR solved a long time ago... if you can talk about it, what's different about your project?

> The tools to do this, do not currently exist.

Considering that all the major networks beam their broadcasts all over the world in real time I don't see what the challenge is other than storage space. Startups like Justin.tv have figured out how to not only capture but stream thousands of simultaneous live streams, so the tools certainly exist.

The primary difficulty is that the Library is attempting more than just live-capture of video streams. They are going for full archival quality. They're not simply saving streams to tape but but running extensive qc processes on everything and transcoding coding the material into multiple formats. Technically, the technology exists, but the Library does not have the hardware to run the software. (Unless someone knows of some efficient, off the shelf content-validation program that we don't know of...)

Other problems include, yes, storage. The projected aggregation rate will outstrip, not only the Libraries current physical facilities, but all planned new facilities, by 2013 (at which point we will be looking at 250PB of storage.) The rate of growth will continue to increase from there. Unless tape technology improves at an exponential rate, this seem unavoidable.

Are you storing everything or just their unique content? Commercials and what not make up a large percentage (30%) and then re-runs (unknown percentage, but huge). And why multiple formats? Unless you're distributing the content it seems more efficient to transcode it when you need it.

I really don't know. I don't make those decisions.

so, you're saying, right now its too expensive.

> I think I'd of heard of this.

Two things are wrong with this statement. The second one is the presumption that just because you work on one project you would know about all the rest: http://www.loc.gov/tweet/how-tweet-it-is.html

Actually, considering how things run around here, it's amazing that I've heard of it at all.

Is anyone concerned about copyright issues? I'd have to go back and consult the Twitter TOS...

edit:

"You agree that this license includes the right for Twitter to make such Content available to other companies, organizations or individuals who partner with Twitter for the syndication, broadcast, distribution or publication of such Content on other media and services, subject to our terms and conditions for such Content use.

Such additional uses by Twitter, or other companies, organizations or individuals who partner with Twitter, may be made with no compensation paid to you with respect to the Content that you submit, post, transmit or otherwise make available through the Services.

We may modify or adapt your Content in order to transmit, display or distribute it over computer networks and in various media and/or make changes to your Content as are necessary to conform and adapt that Content to any requirements or limitations of any networks, devices, services or media."

What copyright issues? The US Copyright Office is part of the Library of Congress. One of the LOC's responsibilities is to maintain a collection of all copyrighted works.

The Library serves as a legal repository for copyright protection and copyright registration, and as the base for the United States Copyright Office. Regardless of whether they register their copyright, all publishers are required to submit two complete copies of their published works to the Library if requested—this requirement is known as mandatory deposit. http://en.wikipedia.org/wiki/Library_of_Congress

Even for non-USA citizens?

Turns out a lot of Tehran-based Twitters are really from elsewhere, such as the U.S. (remember that?).

IANAL, but I don't think your citizenship is an issue. What matters is where a work is published, and in Twitter's case that would appear to be the United States.

I'm no lawyer (especially of international law!), but as far as I know that does not extend to non-American publishers... As a Canadian, what right do they have to archive my tweets?

Is Twitter a non-American publisher?

No, but AFAIK tweets are owned by whoever writes them, not Twitter. You only give Twitter the right to store them, republish them, and so on.

My understanding is that the Berne convention states if you are (for example) Canadian, and your work is published in the United States, then it is covered by United States copyright law.

This has been confirmed on the Library of Congress blog: http://www.loc.gov/tweet/how-tweet-it-is.html

It is not a fake or a late April Fool joke, despite the fact that the original post is from facebook.

I wonder how the content and quality of discourse on Twitter would have differed if its users had known all along that their posts would be archived for posterity by the LoC. This has been like an Ender's Game for hyperbanality!

It's very likely that most people using twitter have no idea what the LoC is.

It will be interesting if they extend their CQL (Contextual Query Language) implementation to support complex searches (e.g. who tweeted first about X)

http://www.loc.gov/standards/sru/index.html

Great, now future generations can see how obsessed we are with Justin Beiber.

How interesting is it that they announced that on Facebook?

One more government database...

First they want to track all of our books... now this? Next it'll be the newspapers. Just you wait.

Books were intended for wide distribution, but not all tweets are.

I don't care that this was done, and it's nice that this snapshot of 2008-2010 will be available for future researchers. But I think this sort of dataset is more useful for nefarious purposes (government investigations) than a bunch of books or newspapers, and it would be too expensive to build for just one investigation.

"Build it and they will come." It's built.

Notice that they said public tweets. This is nothing that someone couldn't scrape off the site with enough time and energy spent.

Technically inaccurate, if you were to start today. Only the last 3,200 tweets of a particular user are discoverable without special access. And that assumes you have a list of all the users.

Ah. I stand corrected. They rest of my point still stands though. This was all information that was broadcast to the world, so there was hardly an expectation of privacy (or at least a reasonable one).

I don't see why this particularly matters. You could likely store the entire twitter archive on a couple of memory sticks, so it doesn't seem to warrant some big library to "house" it.

This is a huge collection of primary sources from lots and lots of people. It's kind of like if you got a copy of every letter sent by people in Philadelphia in the 1770s - most of it would be crap but there would be some pretty neat stuff too.

It's a little different since these are only public tweets. You won't get the same level of private information that you would get if you were grabbing private letters from the mail service.

No. Its fair big data actually. Around 55M tweets per day currently, although it wasn't actually always that big. Maybe that doesn't sound that big, but its at least a few terabytes of data.

I think this is big for a few reasons. One, if its in the LoC then it is (probably) available to the public. Previously it wasn't. That's pretty big.

Also with the LoC having a public archive of it, we can keep better track of politicians who post things online- always a good thing since occasionally they shift their positions or outright lie.

As a data analyst, I'm really excited.

Sure, maybe a few TB uncompressed. Compression would be pretty astoundingly good on that sort of data though - retweets etc.

55 million tweets/day * 140 characters/tweet ~= 14.5 GB/day, presuming a 16-bit character encoding.

That's only about 20 TB for four years worth of data, or about $1500 worth of storage, before compression.

That doesn't seem like it would be a significant challenge for the LOC. I'm more interested in understanding why they would want to use that amount of space to archive Twitter, as opposed to anything else.

Combination of low-hanging fruit plus inflated idea of twitter's usefulness (much like other people largely unfamiliar with today's technology culture)?

There's a good bit of metadata (about another 300 characters or so on average) attached to each tweet that makes it useful. Not a huge amount more, but considering the growth curve... yea 20TB is about right.

Reply on news.ycombinator.com