Internet Archive as a default host-of-record for startups

1) can consume a lot of storage really fast

2) makes IA a bot rather than a user service

3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.

Have you looked at the headers that they send?

  GET /web/20100330210402/https://arxiv.org/abs/0911.1112 HTTP/2
  [...]

  HTTP/2 200 OK
  [...]
  link: <http://arxiv.org/abs/0911.1112>; rel="original",
    <https://web.archive.org/web/timemap/link/http://arxiv.org/abs/0911.1112>;
    rel="timemap"; type="application/link-format",
    <https://web.archive.org/web/http://arxiv.org/abs/0911.1112>;
    rel="timegate",
    <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>;
    rel="first memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT",
    <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>;
    rel="memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT",
    <https://web.archive.org/web/20110101203756/http://arxiv.org/abs/0911.1112>;
    rel="next memento"; datetime="Sat, 01 Jan 2011 20:37:56 GMT",
    <https://web.archive.org/web/20211123040625/https://arxiv.org/abs/0911.1112>;
    rel="last memento"; datetime="Tue, 23 Nov 2021 04:06:25 GMT"
  [...]

Brave browser does that, and I've seen Firefox add-ons that do it. Maybe you can look at their code.

Alternative content addressed systems may also work better for finding the content than someone hosting your DNS records for a long time but the bulk of the problem space is in guaranteeing active hosting in a way viewable to viewers of the age will be available for many years not addressing the content. On top of hosting and addressing the Internet Archive offers the ability to view old content on modern browsers even if modern browsers have 0 support for such content anymore (or if browsers ever had support at all even). Forward compatibility isn't something solved by a protocol.

The Wayback Machine's data is ~20PB..? What is approximately the size of the indexable text (i.e. the text content of html pages, sans tags)? And what would the index size be like, approximately?

I imagine that creating (and maintaining, of course) the index would be the most time-consuming part? Is it at all possible to imagine hosting this index... somewhere... and doing sqlite http range-like queries on it..?

Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is in doc ids 2000, 4000, etc.?

And have separate docid -> archive.org url mapping?

Do you have source/more info than that?

Lets say the internet archive is 100 PB [1], that's 100,000,000 GB [2], and at that rate it comes out to $200 million [3] for the whole thing forever. That's a lot of money, but also a lot less than I was expecting for something like that.

[1] https://www.protocol.com/internet-archive-preserving-future: "The web archive alone is about 45 petabytes — 4,500 terabytes — and the Internet Archive itself is about double that size (the group has other collections, like a huge database of educational films, music and even long-gone software programs)."

[2] https://www.google.com/search?q=100+petabytes+to+gb: "100 petabyte = 1e+8 gigabytes"

[3] https://www.google.com/search?q=1e%2B8+*+%242: "1e+8 * (US$ 2) = 200 million US$"

Edit: I'm assuming they can deliver reliability and durability similar to modern cloud standards, like AWS S3.

I guess archiving a static blog is already trivial for their system, but I'd pay $100 a year to have IA host my static blog. The overhead I would consider as a donation to a worthy cause.

Then again, I can just give them $100 a year and find some free static hosting, like GitHub Pages, and call it a day.

If you had to choose between the GDPR, & an accurate historical record, which would you prefer?

Will you be happy when you've burned down that library?

There are very good reasons that archives will not destroy or alter information outside of very clear difficult and manual processes.

And actually, looking at it, I don't think they're necessarily in violation of GDPR [0].

Point 3 says: "Where personal data are processed for archiving purposes in the public interest, Union or Member State law may provide for derogations from the rights referred to in Articles 15, 16, 18, 19, 20 and 21 subject to the conditions and safeguards referred to in paragraph 1 of this Article in so far as such rights are likely to render impossible or seriously impair the achievement of the specific purposes, and such derogations are necessary for the fulfilment of those purposes."

According to GDPR, national law of EU parties overrules GDPR when it comes to personal data being used in archival context. I don't know every EU country's stance, but most of the bigger economies would allow for this.

There is also a difference between deleting the data and rendering it inaccessible to the public. Keeping something under wraps is generally more 'acceptable', but active destruction of the item (digital or not) and its providence is much more limited. Also there's a difference between personally identifying data (covered by GDPR), your content (which would be covered under copyright and not GDPR), and connections people can make if that content is available (not covered at all because it's not anybody else's issue if you write something terrible and people keep recognizing you over it so long as you did actually write it).

[0] https://gdpr-info.eu/art-89-gdpr/

> Article 3(2), a new feature of the GDPR, creates extraterritorial jurisdiction over companies that have nothing but an internet presence in the EU and offer goods or services to EU residents[1]. While the GDPR requires these companies[2] to follow its data processing rules, it leaves the question of enforcement unanswered. Regulations that cannot be enforced do little to protect the personal data of EU citizens.

> This article discusses how U.S. law affects the enforcement of Article 3(2). In reality, enforcing the GDPR on U.S. companies may be almost impossible. First, the U.S. prohibits enforcing of foreign-country fines. Thus, the EU enforcement power of fines for noncompliance is negligible. Second, enforcing the GDPR through the designated representative can be easily circumvented. Finally, a private lawsuit brought by in the EU may be impossible to enforce under U.S. law.

[snip]

> Currently, there is a hole in the GDPR wall that protects European Union personal data. Even with extraterritorial jurisdiction over U.S. companies with only an internet presence in the EU, the GDPR gives little in the way of tools to enforce it. Fines from supervisory authorities would be stopped by the prohibition on enforcing foreign fines. The company can evade enforcement through a representative simply by not designating one. Finally, private actions may be stalled on issues of personal jurisdiction. If a U.S. company completely disregards the GDPR while targeting customers in the EU, it can use the personal data of EU citizens without much fear of the consequences. While the extraterritorial jurisdiction created by Article 3(2) may have seemed like a good way to solve the problem of foreign companies who do not have a physical presence in the EU, it turns out to be practically useless.

I guess after some time the nodes would agree on the hash and throw away the data because it would cost too much to store.

I see blockchain as a technology that may develop useful applications, but-- in terms of current day usage-- I'm extremely skeptical when it's referenced in conjunction with applications that might achieve the same goals without it.

“Any sufficiently long Internet discussion will propose blockchain as a solution.”

rather than

“Blockchain is eating the world.”

I actually think using blockchain for things like ensuring providence is interesting, since in archives being able to have a clean record of what happened to a piece is VERY useful. It just won't earn a ton of money, so we'll need to wait for the capitalism to burn off to see more not-for-profit uses.

Of course, ensuring that there's persistence of attention as well is a tougher problem. But one only needs to look at sites like https://reddit.com/r/tumblr to realize that there is immense societal interest in "meme archaeology." Reducing the barriers to entry to would-be archaeologists, giving them a "chain" of breadcrumbs that lead to content, and building communities that will socially reward people for their archaeology work, is the best thing we can possibly do.

Concretely, this would be to skip the "many people on encountering a dead URL don't bother to try the internet archive" problem.

https://fleek.co/hosting/

https://docs.textile.io/buckets/

https://www.pinata.cloud/

What is the point of so-called "DNS privacy/Private DNS" if "anyone can tell which site you are visiting" simply by observing IP addresses, without any need to see domainnames.

If SNI (plaintext hostnames sent over the wire) is a non-issue, then why are people working on encrypted Client Hello in TLS1.3.

To get the link to the latest, can use memento. For example,

   usage: echo example.com|1.sh

   #! /bin/sh
   read x;
   curl -A "" https://web.archive.org/web/$(curl -A "" -s "https://web.archive.org/cdx/search/cdx?url=$x&fl=timestamp,original&limit=-1"|tr \\40 /)

That's what interests me. For example, there's a cool repository of 12 step speaker meeting talks hosted in Iceland [1] and frankly, some of the talks are junk, but there's a lot of wisdom. What I find interesting is how it showcases how ordinary citizens talk to each other. The words they use, the accents, the little gems of folk wisdom contained, along with some uncommon stories.

This will be valuable 100 years from now if, for example, you want to build a virtual world based in the mid to late 20th century and you want to get the accents correct. What phrases did people use? What were some common misconceptions? Maybe 100 years from now addiction will no longer be a problem. If my virtual world is to be accurate I need to know what it was like for ordinary people when it was a problem. Etc...

[1] https://xa-speakers.org/

Its fascinating when you start looking into any historical time period (you wouldn't even need to go far back), before a lot of details are educated guesses. Since no one chose to record the mundane in detail or it failed to preserve over time.

"The Internet is forever" isn't some natural law by which all content abides as though it can never disappear. It's a warning that you don't control the content once it's accessible on the Internet.

And requiring an explicit opt-in would basically mean no IA.

To be clear, the IA is a positive, maybe even a great one. But it skirts by because most people don't care. (They did as you say post whatever on the public internet.) Add the facts that they're a non-profit, aren't trying to monetize their hosting, and will generally take things if the owner asks.

Libraries and other archives have some very limited special rights (which mostly relate to making physical backups of physical books). But invoking "library" isn't some general get out of jail free card with respect to copyright.

Your example is an example of choice or consent. They also had the option to burn their hand written books and scrolls down periodically. Systems like IA take that choice away.

Thanks.

that's not static assets, that's a multiuser game server.

i think he's envisioning entire internet service backends that can be packaged up like java applets and re-run on demand paired with some kind of decentralized serving infrastructure that any user can insert coins and resurrect a sophisticated web service from the past.

more likely i suspect we'll see more efforts by hobbyists to resurrect these things and more releases of backends from failed projects into the public domain.

with so much physical gear that requires service backends being made today, we may even see regulation that requires release of the source for a service when a service is shut down. crazy to think that if the company who made your car or tractor fails, that your perfectly good car or tractor could cease function when they shut down the service backend.

I don’t think that is correct. A lot of it was added via automated methods?

I would rather revert to the internet where quality content was published openly on the web. But presently web content, at least that upranked by Google, is very low quality (seo, clickbait, biased, trivial)