Nice. I made something pretty similar here: http://www.wolfire.com/comic
You're afflicted with the non-functioning-hashes-in-history problem though. This fixed it for me: http://github.com/tkyk/jquery-history-plugin
Though, it does get annoying ending up with 100 non-pages in your history. Maybe a big "navigate by hashes: yes/no" option would help us out.