Hacker News Reader

Comment by LeoPanthera | Hacker News Reader

Does archive.org archive flash stuff? Decades into the future, playing back archived flash games and animations will be very difficult.

View on HN

One problem with archive.org is that it retroactively obeys the robots.txt file, even if the files have been spidered and archived [0].

For example, consider the case when a new domain owner attempts to block all bots from spidering their site, by adding something like this to their robots.txt file

    User-agent: *
    Disallow: /

This is actually a fairly common case when domain resellers purchase expired domains.

Now when you try to visit the archived link, because the live robots.txt file disallows bots, you won't be able to access the archived site (which may have been owned by someone completely different).

[0] https://archive.org/post/406632/why-does-the-wayback-machine...

This is true, however they don't purge any of the old archives. If the robots.txt is updated to allow bots again then all of the previous archives will become available again.

it seems weird to me to retroactively obey robots.txt for a _different_ version of the site.

It makes much more sense to archive the robots.txt along with the content, and use the robots.txt linked with that version of the site. Updating the current robots.txt shouldn't affect a past archive.

I believe the reasoning is that if I accidentally published a public page and didn't realize it but later fixed the problem, I'd like to make sure no one else could see it later. It's probably easier than dealing with all kinds of takedown requests.

Yes, archive.org archives Flash and other content, as long as something links to it so the archiver can find it.

Standalone flash players exist, and flash-to-HTML translation will continue to improve.

This is what the Shumway project is for.

https://mozilla.github.io/shumway/

Reply on news.ycombinator.com