This is a problem originally solved by Gauss for Ceres.
https://en.wikipedia.org/wiki/Gauss%27s_method
More generally given sufficient observations one can use many similar methods of orbit determination to get a relatively accurate orbit of the body. All of this is standard stuff in astrodynamics and happens on a near constant basis for both man made and natural objects.
A good introduction is the classic BMW
https://www.amazon.com/Fundamentals-Astrodynamics-Dover-Aero...
None of this requires the shape or material of the object, just some fuzzy measurements of the position
The effects aren't significant for planets, but they are for asteroids. It seemed more significant for Oumuamua than most, suggesting a high surface area to volume ratio.
If I remember right, one of the voyager or pioneer probes was moving much faster than it should've been (relatively), and we had no idea why for years. Then we figured out it was the cumulative effect of heat radiation(?) from something on board the machine.
If we can't figure out what WE made, how can we expect to figure out a once-in-a-generation event like this on the first shot?