Hacker News Reader

But the wayback machine is pretty klunky and slow, honestly.

This is very true. Sometimes it takes 5-10 seconds to load the calendar view for an archived page, and another 5-10, or more, to load a snapshot.

They have a ton of data to manage with limited resources, but it still seems it should be possible to go faster than this. If there's just not enough budget for I/O, maybe they could offer a donate-for-data-dump option, where you can donate in exchange for loading data of interest (say, BBC archives) into a storage medium or query engine of one's choice, so one could do research at a much faster pace.

This is true, but they're also taking on an absolutely monumental task on a shoestring budget. I continue to be amazed at what they are able to accomplish. There aren't many heroes on the internet, but the Internet Archive team qualifies.

Thank you! We're doing what we can with what have where we are, or maybe just what we must because we can, depending on your preferred aphorism :)

Slow, yes. Klunky? Absolutely not. When you make a request to the Internet Archive, you're searching through a massive amount of data. The fact that it only takes a few seconds to pull up a decade-old webpage is amazing.

"But the wayback machine is pretty klunky and slow, honestly."

archive.org is much less annoying if one avoids using a bloated browser and Javascript to make HTTP requests

here is a more lightweight approach, not nearly as klunky/slow, IMO (393 bytes)

    usage 1: (simple html page of all results)
    echo https://www.theatlantic.com/article/619320/|1.sh >1.htm
    firefox ./1.htm

    usage 2: (retrieve last result)
    echo https://www.theatlantic.com/article/619320/|1.sh 1 >2.htm
    firefox ./2.htm

    #!/bin/sh 
    read x0;
    x1=web.archive.org;
    curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original" \
    |case $# in :)
    ;;0)( printf "<h2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$x0</h2><ol><pre>\n";
        sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
        printf "</ol></pre><br>\n" )
    ;;1)curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p);
    esac

haproxy + nc version (965 bytes)

maybe it is faster than curl, maybe not; you be the judge

    #!/bin/sh
    read x0;
    x1=web.archive.org;
    printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms 
    \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b 
    \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \
    |exec haproxy -D -f /dev/stdin;
    printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \
    \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80 \
    |case $# in :)
    ;;0)( printf "<h2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$x0</h2><ol><pre>\n";
        sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
        printf "</ol></pre><br>\n" )
    ;;1) printf "GET %s HTTP/1.1\r\nHost: $x1\r\nConnection: close\r\n\r\n" \
         $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \
         |exec nc -vvn 127.21 80;
    esac;
    if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi

Edit: Remove Wayback Machine's Javascript inserts (usage #2)

curl version

     #!/bin/sh
     read x0;
     x1=web.archive.org;
     curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original"|case $# in :)
     ;;0)( printf "<h2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$x0</h2><ol><pre>\n";
         sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
         printf "</ol></pre><br>\n" )
     ;;1)x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036');
         curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p)|exec tr -d '[\02\03\36]' \
         |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/;
         s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \
         |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;";
     esac

haproxy + nc version

     #!/bin/sh
     read x0;
     x1=web.archive.org;
     printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms 
     \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b 
     \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \
     |exec haproxy -D -f /dev/stdin;
     printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \
     \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80|case $# in :)
     ;;0)( printf "<h2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$x0</h2><ol><pre>\n";
         sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
         printf "</ol></pre><br>\n" )
     ;;1) x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036');
          printf "GET %s HTTP/1.1\r\nHost: ${x1}\r\nConnection: close\r\n\r\n" \
          $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \
          |exec nc -vvn 127.21 80|exec tr -d '[\02\03\36]' \
          |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/;
          s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \
          |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;"
     esac;
     if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi

We've got a pretty solid diff view now, notes at https://blog.archive.org/2019/10/18/the-wayback-machine-figh... and an example at https://web.archive.org/web/diff/20170118202526/201701200403...

Re slowness, we're able to do a lot with a little, but there's always room for improvement. If you're interested in some of the specific infrastructural challenges, I did a presentation in February:

https://archive.org/details/jonah-edwards-presentation

and my colleague did a fantastic presentation detailing some of the internal workings of the Wayback Machine just last week:

https://archive.org/details/bridget-bell-presentation

Do you have statistics on what fraction of the pages on major outlets you have archived, and how often they change?