Some container formats, such as ISO Base Media File Format aka "mpeg4", have features such as edit lists. With edit lists you can have original video in the bitstream, organized in GoPs as you say, but then the final video rendered in the player gets to choose which frames to play from that material.

This allows cutting at completely arbitrary places (I'm 99% sure that this can also be at a higher time resolution than the original data), with the cost that the original (unplayed) data remains in the file.

However, some players may have incomplete support for them. For example I believe FFmpeg only supports edit lists used at the beginning of the video, which is commonly used to align video and audio properly.