There are so many AV setups, we had to leave calibration to the user (we tried with auto-calibration but it could not always be perfect).

The fundamental problem is that there are two independent delays that both depend on your individual system: the delay from the time that the console produces a video frame to the time that the user sees it, and the delay from the time that the console produces a sound to the time that the user hears it. In a beatmatching game, you really need the user's perceptions to be in sync, which means delaying either the video or the audio. Of course, the more you delay one or the other, the more the repercussions you run into.

In a regular video game, it's not a big deal if you fire a gun and hear the shot 50ms later, but in a beatmatching game, that delay is really noticeable.