Thanks for the tip. Where did you get this installer from? I just installed emacs-23.4-bin-i386.zip from
http://ftp.gnu.org/pub/gnu/emacs/windows/ on a Windows VM, and it's a plain zipfile you extract to C:\Program Files (and optionally run addpm.exe for start menu shortcuts) and it just works -- except for things like grep, which require cygwin or equivalent.
Did your installer also install the cygwin (or equivalent) tools?