For anyone looking into this, Rotation Averaging by Hartley et al. (2011) is a must-read on the subject of rotation averaging: https://users.cecs.anu.edu.au/~hartley/Papers/PDF/Hartley-Tr...