Notes on bookmarks from 1997
notes.pinboard.in
notes.pinboard.in
This is a good step, but there's one massive problem with the Internet Archive: If the domain registration lapses (or is sold), the new owner can direct the Internet Archive to remove archived copies of the site previously hosted on that domain[0].
In other words, even if the archives are there today, they may not be there when you want them a year (or more) from now). I've been bitten by this in the past.
This means that the Internet Archive wouldn't have been able to prevent or mitigate some of the cases of link rot described here, as the content could have been scrubbed from the archive.
EDIT In case it wasn't clear, I still support the Internet Archive - I just want to make sure people know this issue exists and that it's not a "silver bullet" for this problem.
[0] I'm fairly certain this is still the case; if they have changed their policy recently I would be pleasantly surprised.
These often have exemptions written into law to allow them to do what they do, so I hope the IA is covered.
You can be trained as an archivist. You can get a Masters and a PhD in archival practice. There are industry-standard procedures and codes of ethics. There's a very specific understanding of what is important to save, how to save it, and how to document its context and its provenance.
That's why the Internet Archive requires a certain fidelity of capture (WARCs) that a screenshot service or a citation tool don't provide.
That's also why they are legally a library. Libraries have particular copyright exemptions for preservation. A typical person doesn't. But you generally have the right to make backups for your own use, and so you can also donate those backups.
It's like if you were a famous person, and you bought a newspaper and a book, and when you died your personal effects were donated to your alma mater who put on a big exhibit of your life and times, that newspaper (your backup of the original that lives in the hard drives of the publisher) and that book (your backup of the original that some author wrote) are there, too. No-one's conferring any rights to the content; the publisher still owns the newspaper, and the author still owns the book, but that was your copy that is now available for everyone to see.
Citation? I'm not aware of the IA having any special status.
A physical newspaper or book isn't a backup. It's a physical artifact that falls under first sale doctrine. The same doesn't apply to digital.
If the random individual is presenting it in the same way as archive.org (namely, citing the source), then I don't see a difference.
That's 6 redacted URLs of 77, versus 35 good ones. I'm not going to not donate to the Internet Archive and help preserve access to those 35 because of those 6.
EDIT: In the interest of rigor, of the 76 404 Not Found URLs, 4 URLs (three domains) had been blocked by a later robots.txt. 45 had relevant content preserved and accessible. That's 10 redactions total.
Also, when the Internet Archive imports third-party captures, like those from Archive Team, they are included irrespective of the robots.txt from the time of capture, which is then only used to manage display of the content.
It's LOCKSS, lots of copies keeps stuff safe. The Internet Archive isn't the only solution; we need more archives. But it's a start, and getting all the bookmarking services to contribute to it, and to improve and make public their caches would be a great improvement.
Would like to hear more about the basis for such a policy, whose only use case seems like censorship of the past, which Archive is clearly not aligned with.
As far as the mechanics of it, they follow a policy that explicitly recommends respecting robots.txt:
https://archive.org/about/faqs.php#Rights
http://www2.sims.berkeley.edu/research/conferences/aps/remov...
But they're still made of cheap paper, and will eventually decompose, or your garage will flood, or bugs will eat them, or mold will be particularly bad one year. And, like anything physical, unless someone makes a pilgrimage to your garage (or that vault), no-one else will ever (re)discover them, or share them. That's the problem with physical things: they can be controlled, hoarded, kept secret, or destroyed.
Born-digital artifacts, and digitized physical ones, can be much more discoverable than physical ones. They often can be repurposed and remixed much more easily. And they absolutely can be copied and distributed widely in ways that physical artifacts can't.
Sticking to paper isn't the answer. Figuring out better ways to index, preserve and distribute digital artifacts is.
The best-case for "born digital" archiving at the moment are probably publications that still produce a full PDF version of each "issue". For example a library could archive The New Inquiry by simply subscribing and downloading each month's subscribers-only PDF, rather than trying to scrape the website. A publication that put the full content in an RSS feed would also be fairly easy to "subscribe" to and archive. But most don't have an easy way of subscribing to full updates that can be archived and stored/viewed independently of the original site, and many are even kind of hostile to the idea.
Journals are another problem. Paper journal subscriptions go into a library's permanent holdings, while digital journal subscriptions often give them cheaper per-issue prices, but in return it's more like renting access to the journal-controlled archive, which goes away if the library cancels the subscription. Most don't allow a library to mass-download the archive into the library's own digital holdings, though some do have some self-archival arrangements (e.g. a library might be allowed to self-archive issues that came out during their period of subscription, but not the whole back catalogue).
I'd go so far as to say that publishers who do not do this and provide these to the Library of Congress should receive much less copyright protection from the government. Copyright and preservation should go hand in hand.
How would this apply to out-of-print books?
If your bookmarking habit is article, rather than web-services, your bookmark manager needs a way of saving bookmarked content.
The interface isn't really designed for using it as an Evernote replacement, but for some use cases at least, it would be workable.
Consider asking Zotero to do full WARC captures, which you could then donate to the Internet Archive and have them be included in the Wayback Machine, which would benefit everyone.
If cited material were automatically archived and submitted to TIA, this could be further useful. The fact that this is an inherent archival of information which is deemed relevant and citable is also worth noting.
We have started crawling the outlinks for every new article and update as they are made – about 5 million new URLs are archived every day. Now we have to figure out how to get archived pages back in to Wikipedia to fix some of those dead links. Kunal Mehta, a Wikipedian from San Jose, recently wrote a protoype bot that can add archived versions to any link in Wikipedia so that when those links are determined to be dead the links can be switched over automatically and continue to work. It will take a while to work this through the process the Wikipedia community of editors uses to approve bots, but that conversation is under way.
I have 28MB of personal bookmark files I'll be post-processing, as well as ~47,000 links from a shared, private bookmarking service dating from 2005 through 2011. I'm not looking forward to it.
Major points:
* Automatically saves an exact copy of pages (no need to explicitly bookmark) in the background.
* Data is encrypted with your password before being sync'd to cloud.
* Search through your pages.
* Works as a Chrome browser extension. No need for a native app.
If you save a page but someone else needs it, they're out of luck.
But, if, in addition to making you a private, encrypted archive, they also tested to see if the URL was publicly visible and, if so, made a WARC of it, then they could package up all those WARCs for donation to the Internet Archive, and everyone could benefit.
There is a sharing feature to solve this problem. :)
But I agree with your point.
I actually looked into WARC earlier but didn't have the bandwidth to do it my first version. When I implement the ability to download your data, I'll try hard to use WARC. Unless there's some brain damage in the format: I hope not! :)
Here are some previous comments with links that might be useful:
A while back, I imagined bringing page content itself—and not just choice metadata like its URL and title—into the purview of the browser's history backend, too, effectively enabling WYGOIWYGO (What You Got Online Is What You Get Offline).
(I started off, funnily enough, not trying to imagine the next logical step in the "moar history" march, but instead with the Internet Archive in mind. I was trying to think of a way that would give ordinary plebs a zero-effort way to add to the Wayback Machine actual archived content in the same way that Alexa and the Internet Archive were slurping in data from the Alexa Toolbar about what pages are getting hits out there.)
After the stuff that happened last January with Aaron Swartz, I was even motivated to write up some use cases and gave it the codename "Permafrost":
Ashley just wants all the content he bookmarks (or simply accesses) to be always available to him, without being frustrated months or years from now by 404s, service shutdowns, and web spiders stretched too thin, allowing his favored content to slip through the cracks of their archiving efforts.
- < https://wiki.mozilla.org/Permafrost >
Even so, it remains one of those projects that I should really get around to kicking off someday, but may never end up starting, much less get close to "completing".
Surely it's obvious that the more copies the better for the backup purpose. Then, why is he advocating cloud services including the Wayback Machine? There's no doubt that it's important for everyone to save web pages locally to create as many copies as possible on the earth to prevent the disaster of the Library of Alexandria from happening again.
You absolutely should save everything yourself. But you should also give copies to as many centralized services as possible. Lots of copies keeps stuff safe.
(BTW: Pinboard is one of my favorite one-person companies, the other being Tarsnap. I love that the both seem to be run be people interested in building something and are not just looking for an exit strategy.)
Bookmarks have never been useful.
Ideally a browser should cache a page when you bookmark it and optionally refresh the cache when you revisit. That way, if you get an error you could look back at the cached version instead.
In fact, that'd be a damn useful browser plugin.
Saving things in a way that those backups can be shared and distributed, like WARC files donated to the Internet Archive, can help ensure that when your hard disk crashes and your backups fail and you lose those PDFs, the things you wanted to save will still be out there.
VRML tutorials are all fundamentally about birth control.