I think !$ takes the last argument on the previous command in history, not previous one on the same line.. at least in bash. For example, if you do:
$ ls /
$ ls /tmp/junk; rm !$
It's taking "/" rather than "/tmp/junk"..EDIT: formatting