Interesting idea. I've the bash built-in "pushd" and "popd" for a long time for similar reasons, but those effectively limit you to treating the history as a stack rather than an arbitrary list.
e.g., a command like "tar cfz bak.tgz ~3 ~2/subdirectory ~1" would mean something else once you've "pushd +3".
I always find it hard to undo my sequence changes to get the stack in the same order as before, so usually end up typing "dirs -v" a lot (or its alias) and renumbering my ~N references.
I have ported this functionality to Bash: http://thrysoee.dk/pushd/