I have experienced the same, but after updating raspbmc to RC5 it runs just fine.