I'm not saying remove the possibility of 32-bit userspace and nor are, it seems, freebsd. I am saying don't bother with a 32-bit kernel. Otherwise you're limited to a 4GB address space and hacks like PAE. No harm in running 32-bit binaries if you want and, I think Solaris actually left userspace as 32-bit only even on their 64-bit OS, unless the application really benefited from a 64-bit setup. Not sure what they do now, been a while since I used it.
It's arguable ASLR only works well in a 64-bit mode, as well.