Hacker News Reader

Comment by cyphar | Hacker News Reader

URLTeam (part of the larger ArchiveTeam) are doing this for quite a few different URL shortening services and they're providing torrents to download the database[1] (similar to how they created torrents for the entirety of GeoCities[2]).

I would recommend contributing to their various tools[3] so that the archive can be maintained for future digital archeologists.

The Wayback Machine is run by the Internet Archive (and ArchiveTeam is a project run in part by folks from the Internet Archive and a whole gaggle of volunteers), so I'm not really surprised they were happy with you providing backups of URL shortened links. But I would recommend helping URLTeam more directly (since their method of keeping archives of URL shorteners is much more efficient).

[1]: http://urlte.am/ [2]: https://thepiratebay.org/torrent/5923737/Geocities_-_The_Tor... [3]: https://github.com/ArchiveTeam/terroroftinytown

View on HN

ArchiveTeam and URLTeam do great work, but those projects and the OP have different goals. (I'm the OP.) This bot will NOT produce any kind of corpus of tweeted links or links shortened by Twitter's t.co shortener; in fact, it bypasses t.co entirely and I don't have any kind of record of those shortened links.

Instead, it backs up the contents of the pages that are linked to at the time of the tweets. Frankly my tool doesn't do anything interesting at all with the URLs — it just submits the "expanded URL" that was tweeted to the Wayback Machine and lets it sort out any and all 301s.

Right, but my point was that resolving the 301 and storing the result is more efficient overall than doing a full mirror of a URL only to find out that it's an alternative URL with their tracking stuff appended to the end of the URL.

But yeah, if you want to bypass t.co (meaning that it's not useful for archeologists) it makes sense to just archive them.

It might be nice to save as well:

(1) the tweet-detail page, for the tweet that includes the link;

(2) the t.co mapping, so that the tweet-detail page's t.co link can somehow be resolved to the (archived) page to which it links.

I don't think there are any blocks against doing (1).

Unfortunately for (2), Twitter has a blanket robots.txt prohibition in place for domain t.co. Perhaps IA could be convinced to ignore that robots.txt in the public interest.

Alternatively, perhaps another site could be set up that itself accepts t.co link-paths, in the background queries t.co, and returns both an HTML page and working redirect that isn't robots.txt-blocked. LinkArchiver (and any other similar sites) could as a convention archive responses of this other site whenever they'd like to archive t.co.

In the case of the developer, contributing to an open source project might not meet their goal quite as well. The immediate purpose of the project was to improve their programming skill as part of their time at the Recurse Center. Working on a large open source project flexes a different set of muscles.

Reply on news.ycombinator.com