I have 4GB system and 512GB video memory installed. However, I only upgraded from 2GB after installation because the already-heavy Java apps I use (Eclipse, SQL developer) now required twice the memory.

As the other commenter mentioned, there are the CPU optimizations to consider, but what decided my final transition (I've tried it out a few times over the past few years) was when System76 moved to shipping their systems with the 64-bit distro installed. I want to maximize their ability to support me, so when I did a fresh install on a new drive I opted for 64-bit as well.

Most of the problems of compatibility with 32-bit binaries have been trivialized or are routed around easily enough these days. Though 64-bit Linux was never as much hassle as x64 Windows. ;^)