Hacker News Reader

Comment by user5994461 | Hacker News Reader

I had to decom a major wiki in a very large company not long ago. I can give you four reasons why these things happen, possibly without notice.

1) The wiki software and database have been abandoned for years. There is no maintenance and no further release.

2) It will stop working shortly. Like, it doesn't start on Ubuntu > 14 or current MySql at all.

3) It already stopped working and/or the content is already lost. Can be an accidental deletion or the 10 year old server passed away.

4) Security vulnerability. Remote code execution / SQL injection in the wiki software. That can't be fixed because point one.

I wrote a longer blog post on software death cycle in companies https://thehftguy.com/2019/12/10/why-products-are-shutdown-t...

View on HN

It shouldn't be hard to save a static copy of the actual HTML and replace the dynamic site with that though. Or at least give others enough time to do the same.

Not that hard?

You'd need to process each page then data mine it.

A company that can build a game engine should probably be able to crawl a site and save an html dump?

Just archive.org every page on the wiki. Their api will let you do 15/minute.

HTTrack (https://www.httrack.com/) makes tasks like this trivial.

It doesn't run a full browser engine so it won't work with the vast majority of websites.

It's not hard, except when it is and it takes weeks of work. From my own experience:

1) httrack wasn't able to crawl anything for some reasons. wget only with special flags available in the latest version.

2) All the links are broken. Wiki have their own linking system between pages that are processed on the fly. The archive with all links hardcoded to wikidump.example.com/wikidump/page really doesn't work in place.

3) More challenges with saving pictures and attachments.

4) Unicode characters in some pages, that broke the page and/or the crawler.

5) Infinite crawling and a ton of useless pages. Consider that every URL with a query string is a separate page. /page?version=50&comparewith=49

6) Crawling large amount of documents takes forever. Could be an hour wasted on each try. Consider tens or hundreds of thousands of unique URLs to save, see point five. Really wish the crawler could parallelize and run on the same host as the wiki.

Good points. Perhaps I should have said, "It should be very possible," or similar instead. I'm sure there would be challenges, but I would expect them to be surmountable.

Reply on news.ycombinator.com