You're correct about that there are different degrees of seeking support.

Seeking VBR MP3 with perfect accuracy is trivial by forward-reading. However, this is obviously highly inefficient on long duration seeks.

For instant seeking support, you need to (partly) depend on the optional VBR headers. This comes with its own set of issues, e.g., the most commonly used Xing header contains only 100 seek table entries, which may not provide enough resolution for large files.

I'm still surprised about the complete lack of support for those headers in AVFoundation, since I would consider it a low-hanging fruit in terms of improving usability for the majority of use cases (excluding pod casts).

Disclaimer: I've worked on MP3TrackDemuxer for Gecko/Firefox.