Offhand, one of the big issues that comes to mind is latency. DDR4 has a much higher latency than DDR2 between requesting and receiving, so could be potentially slower in a simpler core that doesn't have as big of a cache, and less aggressive cacheing/fetching algorithms.