I'd say you waste time taking a conscious decision between 64-bit and 32-bit, there are few good reasons to not just go with 64-bit, and dwelling on it just wastes time. Do you have any metrics to support your claim that 32-bit Debian is faster than the exact same 64-bit distro?
And what difference does virtual versus physical machines make? Heck, you can even run virtual 64-bit guests on a 32-bit host machine if the CPU supports either Intel-VT or AMD-V.