Youtube is more distributed, has more hardware, more available bandwidth and can be served from closer to you. So probably they can't change that quickly. But the whole experience is probably related to your location. It's not the fastest service around, but for me trailers stream faster than the playback speed. (I'm guesstimating the size of their network since the was the only thing they did as youtube itself, then google could only add to that - does amazon has enough spare capacitly to top that? I don't know, but wouldn't expect it - not as distributed as google at least)
Maybe you're just too far from them (from the network topology point of view).