This is just describing SLAM (https://en.wikipedia.org/wiki/Simultaneous_localization_and_... it's a deep field and there are many techniques and tools.

Extrinsic calibration (where is my camera in 3D space, and how is it oriented?) being different from intrinsic calibration (how do the pixels in my image map to ray directions relative to the front nodal point of the lens?).