Internet Data Is Rotting
theconversation.com
theconversation.com
The next best thing would be to redefine what "bookmarking" is. When I bookmark a page, I want it to be permanently stored on my local machine and full-text indexed. In fact, it's rather ridiculous that after 25 years browsers don't have anything of this sort. Unfortunately, the most popular browser in the world is controlled by the same people who control our protocols.
If I ever get the energy, I will attempt to write a browser extension for this.
It needs to be seamless. It needs to be searchable. It would be incredibly useful if it would capture relationships between pages (links) in addition to pages themselves to navigate offline.
I can see many use cases for a kind of "super bookmark mode" where the browser automatically stores all the pages you visit within a certain domain.
You can even take it a step further and create a viewable timeline of all the pages you've visited in case you have something at the tip of your tongue, but need to retrace your steps and logic to get there. Browsing history is kinda lackluster for this. My last ten entries on firefox are three hackernews articles with "Add Comment" dispersed half a dozen times in that list. If I had something along the lines of zoteros timeline for papers I could probably find stuff way easier.
AFAIK there’s no self hosted equivalent. Ive been using it for years. Whenever I put a link in my docs / notes I use that site to cache a copy as well.
When I first signed up I wrote a script to convert my bookmarks to a text list so I could import them. Around 25% of the pages I’d bookmarked were gone.
You can export (with some pain) your Pocket list. Fetching sources from there is ... somewhat painful. I've done it a couple of times, but it's not something I relish doing.
Seems like someone I can trust not to sell out and shut down.
Storage must be local, physicslly user-controlled. All online stuff is a backup only.
Any centralized online site that aggregates other data has an, at best, unclear future ahead of it. Really though, centralized is the key word. The internet so desperately needs to migrate to decentralization but we're collectively about as inertial as an 882 foot long, 46,328 ton tanker, built in 1909.
revenue looks like hes doing just fine. https://blog.pinboard.in/2017/07/eight_years_of_victory/
Chromium is almost there.
Chrome had “chrome://flags/#show-saved-copy” which would do you want if you could make Chrome's cache arbitrarily large and persistent, but it appears to be gone now.
Perhaps I should try to resurrect my rusty and incomplete Firefox add-on skills.
It could easily download entire sites or just a subtree, including resources. It would the rewrite URLs as necessary to point to the downloaded documents instead of pointing to absolute URLs on tje Internet.
Still hoping that Mozilla will improve its extension API. I mean I can see the reason for not allowing everyone to pone around everywhere under the hood, but the current API is rather anemic it seems.
Looking for an alternative that would import my thousands of saved pages.
> Universities, governments and scientific societies are struggling to preserve scientific data in a hodgepodge of archives, such as the U.K.‘s Digital Preservation Coalition, MetaArchive, or the now-disbanded collaborative Digital Preservation Network.
Like any conservation work, the benefits are incredibly easy to ignore - until something goes awry / stops getting funding and suddenly it's too late. Consequently it's easy to have a myopic view of the issue.
These organizations are doing very important work, and I hope that internet users and governments don't take them for granted.
The Internet Archive is doing God's work, but it's not enough. If you don't have the URL of a site that is gone, you probably won't find any reference to it after every online hyperlink to it has disappeared as well. It might become then inaccessible after a while, stored yet gone anyway.
I'd also like to see a distinction between the idea of web servers, which are really publishers, and where archives are kept. Ideally not all in one single store, a/k/a the Internet Archive, but replicated fairly widely.
The effort has been made. Look at what happened with Tibet:
> Caroe pulled off a rather notorious subterfuge in order to buttress the British claim to Tawang: he published the Simla Convention for the first time in 1938 with a note misrepresenting that it had included settlement of the border (and alienation of Tawang); and he arranged for the publication of official Survey of India maps that, for the first time, showed the McMahon Line as the official boundary. To advance the narrative, he also corresponded with commercial atlas publishers to put the McMahon Line on their maps as well.
> In a telling indication of Caroe’s jiggery-pokery, to avoid the awkward question of why he was first publishing the Simla Convention twenty-four years after the fact in 1938, he instead arranged for the surreptitious printing of a spurious back-dated edition of Aitchison, deleting the original note about the Chinese government’s non-signature, and replacing it with a lengthy note stating, quite falsely, that “The [Simla] Convention included a definition of boundaries…”
> Since 1) the McMahon Line had been concluded in secret bilateral negotiations between Tibet and Great Britain outside the Convention and 2) the Chinese had officially refused to recognize any bilateral agreement, boundary or otherwise, between Tibet and Great Britain and 3) had declined to sign the Simla Convention itself and 4) had notified Great Britain in 1914 that the specific sticking point was “the boundaries” this was hoo-hah.
> The replacement copy was distributed to various libraries with instructions to withdraw and destroy the original edition.
> The subterfuge was only discovered in 1963 when J.A. Addis, a British diplomat, discovered a surviving copy of the original edition at Harvard and compared it to Caroe’s version.
( http://www.unz.com/plee/the-myth-of-the-mcmahon-line/ )
Wikipedia confirms this, if you look hard, in a shockingly non-judgmental way:
> Simla was initially rejected by the Government of India as incompatible with the 1907 Anglo-Russian Convention. The official treaty record, C.U. Aitchison's A Collection of Treaties, was published with a note stating that no binding agreement had been reached at Simla. Since the condition (agreement with China) specified by the accord was not met, the Tibetan government didn't agree with the McMahon Line.
> The Anglo-Russian Convention was renounced by Russia and Britain jointly in 1921, but the McMahon Line was forgotten until 1935, when interest was revived by civil service officer Olaf Caroe. The Survey of India published a map showing the McMahon Line as the official boundary in 1937. In 1938, the British published the Simla Convention in Aitchison's Treaties. A volume published earlier was recalled from libraries and replaced with a volume that includes the Simla Convention together with an editor's note stating that Tibet and Britain, but not China, accepted the agreement as binding. The replacement volume has a false 1929 publication date.
Baidu Tieba, which could be considered as Reddit for China, just made all posts before 2017-01-01 inaccessible. And a number of other online forums are doing the same thing due to political reasons.
Project's still there, although not sure how well it does with the modern web.
http://www.gedanken.org.uk/software/wwwoffle/
There are a bunch of more modern variations too:
https://archivebox.io/ - "Your own personal internet archive"
https://getpolarized.io/ (as seen on HN previously)
I think we're due for the idea that a proxy can be designated as a trusted intermediary, most especially if it's run on a personal basis. I'm sure this presents security issues, but it also avoids some.
We have that idea now; you designate the proxy as a trusted intermediary by accepting its certificate. The chain looks something like this:
You: browser, take me to https://youtube.com
Browser: proxy, get me https://youtube.com
Proxy: YouTube, get me /
YouTube: I'm youtube.com -- here's a certificate signed
by the government of Egypt that proves it. And
here are the contents of /
Proxy (to browser): I'm youtube.com -- here's a
self-signed certificate attesting to
that. And here are the contents of /
Browser (to user): SECURITY ALERT! SECURITY ALERT!
https://comodosslstore.com/blog/google-chrome-63-will-warn-y...
If you properly configure your own CA, then the TLS error triggering this behavior won't occur, and there is no security problem for Chrome to put its foot down on -- your proxy is providing a valid certificate for whatever domain, as far as Chrome is concerned, not an invalid one.
Compare https://support.portswigger.net/customer/portal/articles/178... .
> The Chrome browser picks up the certificate trust store from your host computer. By installing Burp's CA certificate in your computer’s built-in browser (e.g. Internet Explorer on Windows, or Safari on OS X), Chrome will automatically make use of the certificate.
> When the Burp CA certificate has been installed for your built-in browser, restart Chrome and you should be able to visit any HTTPS URL via Burp without any security warnings.
I'm glad that data is rotting on the Internet. In fact, I'll go one step farther and say that data should rot more quickly on the Internet. There's no reason that some scummy marketing algorithm should have access to my high-school social media posts. There's no reason that governments or private individuals with a grudge ought to be able to through the history of everything I've written, no matter how off-the-cuff, in order to dredge up something that makes me look like an undesirable when it's taken out of context.
If an individual wants to save a particular piece of information, that should be their choice. Otherwise, by default, information ought to disappear from the Internet.
Managing what data you keep is just too time consuming. Hoarding it all is simpler.
This issue in particular we have great solutions for (open formats / text), but they are of course less profitable than only-my-app-can-read-this formats.
I count myself lucky I was introduced to the HTTRACK archiver program many years ago and thus have complete offline copies of many of my favorite websites of the early 00's.
https://blogs.loc.gov/thesignal/2011/11/the-average-lifespan...
Think of the privilege you had when you were younger. You could do something stupid and nobody could whip out a high def camera to record it and make it part of your history forever.
Let it rot.
As the article points out, a more concerning issue is that “universities, governments and scientific societies are struggling to preserve scientific data”.
Such is life?
For example I have recordings of the Colbert report going back to ~2005. Some of his skits released during that time would be classified as "hate speech" in 2019. Of course he, and mainstream broadcasting companies would love it if you didn't think about that. There are plenty of news clips and interviews where mainstream politicians (on Left AND Right) casually dismiss gay marriage. Powerful tech influencers like Mark Zuckerberg would love it if his IMs would disappear from the Internet. The examples go on and on.
At this point, I'm not really comfortable with what we're preserving.
Our history is an inseparable part of who we are, and it should not be forgotten.
That being said, there's a lot of valuable information on the internet that is absolutely worth preserving. Scrape off the layer of social media and the internet is still a place of learning and problem solving. I do my own repairs on my car and the amount of times I've found a well written photo essay on a particular fix in a random honda forum, only to find the linked pictures broken, is astounding and a shame to say the least.
Kind of like saying “People who go to prison just shouldn’t commit crimes.” I don’t believe in making a case equivalence here.
Being publicly prosecuted has just as much to do with “who” you are as “what” you did. I’m personally not fine with this distinction.
One segment from the final episode featured on HN a few weeks ago. It references the burning of witches, in Scotland, from earlier in the same episode. Burke's point was that a set of beliefs which had once carried legal force no longer does. That route can run in reverse, and it's quite possible for beliefs, or actions, previously legally sanctioned (or at least not criminalised) to become otherwise. Prohibitions against ex post facto laws limit this in many jurisdictions, but changes in social mores are less forgiving. I've lived long enough to see some remarkable changes along several such lines, and have witnessed social and political upheavals elsewhere that are even more pronounced.
There's the question of whether or not it's the fact of a durable record, or the abuse of power through that record, that is the real root of the problem. As Aesop noted, any excuse will do for a tyrant. But the persistent record does seem to pose certain problems. Particularly against a shifting set of judgements.
Things like HP Lovecraft renouncing racism is only preserved because he wrote so many damn letters. It doesn't make the racist stuff he wrote less racist, but it allows a really clear view on how worldviews could change at the time. Perhaps the things we are storing will have historic or philosophical value.
I miss real privacy though.
Yeah, I'm going to need some proof for that. Eddie Murphy's '80s HBO standup routines are available in full on YouTube, and some of that material is absurdly homophobic. He isn't a pariah by any means.
People do see nuance and understand that values and mores change over time, and what might have been socially acceptable at one point, is no longer so.
> GLAAD: Kevin Hart ‘Shouldn’t Have Stepped Down’ as Oscars Host, but Used Gig to Bring Unity
https://www.indiewire.com/2018/12/glaad-kevin-hart-oscars-ho...
If it's just going to be used as witchhunt fodder however than you can fuck right off with that.
Sanity check for 2019:
* "hate speech" should be a link
* when I click it, it should accordion out into links to recordings of Colbert Report that you classified as "hate speech" queued up to the relevant skits
* when I click one it should start streaming the data from all the peers who care about preserving old Colbert Report videos
* Copyright law should have changed a decade ago to protect people who rebroadcasting exact copies of data that the copyright owner/licensee initially publicly broadcast (where "exact" is verified by checking hashes)
So let's think about what we can do for the generation that is about to be born.
> Think of the privilege you had when you were younger. You could do something stupid and nobody could whip out a high def camera to record it and make it part of your history forever.
This is an excellent point, and I agree 100%. But it's not a priori obvious that we need to embrace internet rot to solve this problem. Perhaps we can work towards a future where privacy rights and digital literacy protect individuals from offense archaeology, and at the same time, work to preserve the knowledge, ideas, and sheer human weirdness that get posted every day (as long as the creators don't want it removed. My understanding is that the Internet Archive has explicitly stated that they are not interested in preserving stuff where there has been a takedown request.)
One might argue that we have to make a choice. This is touched on in the article - redundancy presents a tradeoff between persistence benefits and security/privacy risks. But we should examine this tradeoff more closely and see if there isn't a healthy middle ground before we ask for one side wholesale.
Some reddit users are egregious about it, installing scripts that overwrite their comments after x amount of time, seemingly oblivious to the fact that every edit on reddit can be found through archival tools. The solution is to take better care to not conflate your anonymous online persona to your real life persona—just don't post identifying information publicly online and you will be head and shoulders above many users on the internet in terms of privacy. There's no need to purge the internet of it's collective knowledge and history.
We are very lucky to have the wayback machine preserving this stuff from dissapearing into the void, but it doesn't cache everything on the internet, especially if that forum I visited had shut down and became impossible to find in a search result.
Side note: Is there an extension or bookmarklet available to automatically pull a web archive?
Correct, useful, or noteworthy information is often re-posted and copied, leading to preservation.
There is value in archiving things beyond just relying on the small % of things that get actively reposted elsewhere. A good local caching solution of everything you've viewed on the web would be extremely useful for your personal reference, especially when sites frequently close and links die.
If you really wanted to fantasize you could seed your cached site data using bittorrent to maintain a common web archive with everyone on the internet, free from a reliance of one lumbering archival corporate entity that might disappear on the whim of a VC or shareholder, free from commercial profiteering practices, free from the ad men and anyone else who's job it is to pollute and dilute technology for profit, and firmly in the hands of the users of the internet.
I've recently added two to my browser, "open in Wayback Machine" and "Save in Wayback Machine". Respectively:
javascript:void(window.open('https://web.archive.org/web/*/'+location.href));
javascript:void(window.open('https://web.archive.org/save/'+location.href));
(These are from the Internet Archive, not my work.)
For bulk archival, lists of URLs can be submitted to the IA's save address:
https://web.archive.org/save/<URL>
This can be automated with a simple shell script using any console or script-based HTTP agent, such as curl, wget, lynx, etc.
I have 10 years of lovingly curated YouTube videos playlists, which, now when I look into the older ones, are a barren wasteland of "Video removed" or "Video not available". It is heartbreaking. Is there any way I can prevent this from happening?
Check out its options here: https://github.com/ytdl-org/youtube-dl/blob/master/README.md...
With --add-metadata you can embed the YT video description in the video file. The downloaded video file name will contain the YT identifier so you can still match them back if needed.
There is another option to save the metadata to a separate JSON file if you prefer that.
To download your playlists, give it each playlist’s URL instead of the video url:
youtube-dl --add-metadata --ignore-errors 'https://www.youtube.com/watch?v=8GW6sLrK40k&list=RDQMc4l8l2aQrNo'
The option --ignore-errors will jump over the unavailable videos instead of stopping.
Edit to add: If you want to download your playlists as separate directories, with each video file name including its original index in the playlist, see these examples in youtube-dl’s documentation: https://github.com/ytdl-org/youtube-dl/blob/master/README.md...
It's a fun experiment with clear, practical use.
Trying the trial now.
Thanks for bringing it to my attention.
As one of the people helping coordinate information among those still using the platform and hoping to migrate off of it, discovering the Archive Team's GoogleMinus project this past January was a huge boost. That ended up being the largest archival project undertaken to date, 1.6 PB, and succeeded in capturing 98% of all G+ profiles, now stored at the Internet Archive.[1]
While it had long been obvious that the project was ill-stared, the shutdown announcement came as a surprise, and Google's tools, communications, and support for both individuals, and far more importantly, groups, looking to continue their existence off the platform, was abysmal.
I don't fault Google for killing the service -- I was suprised it survived as long as it had. I do fault Google for how they did so. And that episode was hardly the worst in history.
One of the lesser-known parts of G+ were its Communities. In the process of the shutdown we came to realise that there were over 8 million of these, about 50,000 with 1,000 or more members, of all descriptions. Many frivilous or worse, but many also not. And all stuck in a very hard spot by Google's actions.[2]
Even preservation of individual data does very little for groups, and is one of the issues we're considering in the post mortem of the G+ mass migration, intended to be of use to others.[3]
________________________________
Notes:
1. For those preferring not to have their content archived, the IA WBM respect DMCA requests, and as Google+ posts are all listed under the user's account, requesting removal is exceedingly straightforward.
2. Characteristics of number and size are collected here, compiled by me, based in part on on data provided by Friends+Me: https://social.antefriguserat.de/index.php/Migrating_Google%...
3. Discussion at Reddit and elsewhere. Compilation at the PlexodusWiki. https://old.reddit.com/r/plexodus/comments/boa97x/g_migratio... https://social.antefriguserat.de/index.php/G%2B_Migration_Po...
A good history preservation should allow you to somehow "browse" it, as if the historical system is still alive. How the website worked, how it was used, that's all parts of the history. If old operating systems and programs are preserved, there is no reason not to preserve websites in this way.
Back in the old days, many systems are federated and/or distributed, which means the software and the protocol are two separate entities. You use a newsreader, which talks the NNTP protocol to obtain news from a Usenet newsgroup. If you want to preserve history, you can (a) archive the newsreader program with source code, and (b) archive all the data on the NNTP server. That's exactly what has been done already, if you load a Usenet archive to your newsreader, pretty much you would have the experience similar to how Larry Wall browsed the Usenet back in the late 80s, at worst you need to rewrite a compatible "mock" server, but that's all. On the other hand, little of the early BBS systems have been preserved, once the server is gone, everything is gone.
The transformation to the web, means now the platform (a web community) = protocol (backend database format) = user interface (HTML/CSS), they're all tightly coupled together. It creates several problems:
(1) The "internal state" cannot be archived. A website is a system with constantly updating parameters, and often they are not stored. Simple examples: (a) On Hacker News, I cannot see what was shown on the frontpage yesterday retroactively, (b) A user changes his/her avatar, now we had no idea how the old avatar used to look like, and (c) an early user has been banned from the forum, now his/her personal profile is inaccessible, (d) on some social media platforms, sometimes a old post may be raised from the dead by renewed interests (look, how stupid this comment was!), and now suddenly it's flooded by new posts, leaving no trace of how it used to look like.
(2) The "reader/user interface" cannot be archived. You must have seen something like this: You changed the website frontend, superficially, lots of "conservative" users complained, but the point is: now the old frontend and its "look-and-feel" is lost. If it was a simple CSS file, there are chances to bring it back, but if it was a major rewrite of frontend code, now history is gone forever. And in the lifetime of a website, the design and architecture is likely to be changed many times.
As a result, even if a website and all its content is still alive, it may already be a shadow of its past for a long time, don't even mention to preserve it! And currently, there are two ways to archive the web, both are flawed:
(1) Preserve the HTML at the surface. It's good for single pages, but you cannot browse a website in this way at all. None of the button on the website would work.
(2) Preserve the database. For example, using the API to save posts, or dumping the database - the frontend and reader are not preserved. Using Hacker News as an example, now every single post is archived, but it's far from a full experience, at least you should be able to click someone's username and see all the posts.
Now more and more websites are powered by JavaScript, makes the problem even worse. You are now literally running a program on your computer without any control over it. Once the platform is gone, no archive can save you.
What is the solution? I guess there's no full solution, but there are some possibilities:
(1) Wikipedia-like websites already have builtin version control, but it's very difficult to browse the historical version of the entire website. Systems like this can improve the frontend / user interface to allow a user to "lock on" a historical date.
(2) When building an all-Javascript website, spend some energy to build a plain HTML version as well, it may help avoiding the upcoming digital dark age.
(3) If you are going to close a website, perhaps it may be a good idea to make your internal backups of database and codebase at different years publicly available with sensitive information removed, and allow everyone to setup and run a replicated version. It's infeasible for a big website, but it may be a workable idea for a small community.
And I can imagine the archeologist from the 22th century digging into the old backup tapes of Reddit and attempt to rerun the system.
But ultimately, it's a problem that is needed to be addressed by the protocol and software with archive and preservation in mind.
---
BTW: A few weeks ago, I've written a lengthy comment on the fundamental conflict of history preservation and personal privacy, using the Usenet as an example, you may find it interesting.
For the whitewashing thing, it will happen anyway. Only vigilance can protect against rewriting. Websites can be altered. There is no provenance.
I'm not convinced infinite recall is useful.
We've been mourning the loss of the Library of Alexandria for circa 1,500 years and I can't notice that we're getting radically wiser during the last 25 ones.
Letting all go is definitely the cheaper and convenient attitude... for us. But we might be leaving nothing to build upon for the future generation. They should have the same opportunity to ignore what they want that we have.