> The -m32 option sets int, long, and pointer types to 32 bits, and generates code that runs on any i386 system.
That restricts you to the smaller register set, removes all amd64 instructions, and uses the x86 ABI (on Linux, parameters are passed on the stack instead of in registers.) I suspect that this is why your program became slower, as opposed to using 32-bit pointers.