VSLAM, visual SLAM, which is what you have to employ if relying largely on vision-only, is so far cruder than that.
I'm sure Tesla has some sensor fusion that also employs GPS and short-range radar/ultrasonar to help.
You can get some velocity and positional data through GPS, but it will have quite a lot of latency and is very coarse (at least the commercially accessible part).
SLAM is much, much easier to do with LIDAR since that will give you a point cloud 3D model. With VSLAM, you have to infer the depth data and build it up over time, and it's a much harder problem to solve reliably. OTOH, LIDAR's Achilles heel is, however, heavy rain/sleet/snow. Waymo has shown off using deep learning to filter the bulk of the noise out, but it looks like it's still early days there.
Vision isn't great for sleet/snow either, to be fair.
[1]: simultaneous localization and mapping