Um, Quicktime? iTunes? Apple's software was always terrible (as a Windows user).
There is a technical reason for this, and it's not a QT program problem, but rather a QT container problem. The QT container file does a horrendous job at synchronizing audio and visual streams. QT playback software exploits this problem, though, by intentionally lagging the start of a video by a few frames in order for the audio and visual streams to match up on the timeline. By not having to track the sync after the initial playback lag, the file plays more "reliably" and "quickly" but this also means that the decoder has no reference point from which it can scrub backwards. Because the "quick" in QuickTime really is a misnomer. This is lazy time, not quick time.
You are probably referring to the delay introduced by b-frames, but the mov container has got a atom ('cslg') to store the max and min offsets and put everything in sync again.
Unfortunately third party mov demuxers don't support cslg or edit lists, so they only supports the simplest mov files.
AAC, like MP3, introduces a padding of silence at the beginning of the stream. Because modern QT container files do not compensate for this, all audio and video streams within this type of QT file will be off sync by default. QT playback software waits for the audio stream to begin (waits for silence padding to end) before video playback begins, even though the streams themselves line up 1:1 in the container file. This is lazy engineering, not an advanced feature.