That's why Power6 has enormous memory bandwidth (~50 GB/s theoretical).
The theoretical processor bandwidth is one thing; the actual throughput on motherboards with existing slow memory chips (DDR2-667 according to the specs page) is quite a different matter.
This becomes a pretty big deal when you have that many cores all hammering on it at once. Divide by 32...
Sure, so maybe in reality you only get ~30 GB/s/socket, which is 15 GB/s/core. That's a lot more bandwidth than other systems have.
Unfortunately the limit is going to be in the chipsets for that DDR2-667 RAM. PC2 5300 -- that is not a wide bus, 5.3 GB/s. That's the bottleneck.