So we can determine the orbit using only line of sight measurements of the object relative to the Earth.

This is a problem originally solved by Gauss for Ceres.

https://en.wikipedia.org/wiki/Gauss%27s_method

More generally given sufficient observations one can use many similar methods of orbit determination to get a relatively accurate orbit of the body. All of this is standard stuff in astrodynamics and happens on a near constant basis for both man made and natural objects.

A good introduction is the classic BMW

https://www.amazon.com/Fundamentals-Astrodynamics-Dover-Aero...

None of this requires the shape or material of the object, just some fuzzy measurements of the position