A backhoe hooked up to a virtual reality input arm with force feedback was built several decades ago. The operator could feel obstructions, and dig out the dirt around a pipe by feel. Never became a product, though. Probably too early.
I've know backhoe operators who strongly preferred older equipment, when its too new the pins are too tight and they have no "slop" to work with. That slop is what they're using for sensory input, if you watch them. With attention and experience that's real time feedback.
The whole machine is the force feedback mechanism. You're not just sitting there in a comfy chair pulling levers, you're on top of a collection of systems all singing their own song in response to the varying loads placed on them.
Luckily, hydraulics offer a very easy way to do force feedback: Just measure the post-valve cylinder pressure.
That means you can have all your wiring and sensors in the cab far away from the dirt and rain. Makes force feedback pretty easy and cheap.
The human brain already automates the kinematics.
There isn't much upside to automation and a lot of downside because of the diverse instructions required.
https://www.komatsu.eu/en/Komatsu-Intelligent-Machine-Contro...
It's nowhere near autonomous, but it's several times faster than total manual control would be.
Not so much these days, but the general idea is correct.
Usually there are joysticks (one per hand) that provide control over one movement per axis. So for example the right hand joystick on a mini-excavator usually controls bucket curl (left/right) and boom movement (forward/back). Left hand does boom up/down and rotate left/right. There's an ISO standard for this UI. Old school tractor-loader-backhoe machines (what people in the US call "a backhoe") had one lever per cylinder, so more complicated to control and more levers. An old-style motor grader is the extreme example of lever confusion[1].
Operation of heavy machines relies (still) on the brain's capability to virtualize the movement of the machine in terms of the body's limbs. That's why you need some time training to become proficient. After a few hours you just think "I want the bucket over there" and your hands make the necessary movements without thinking.
There's already lots of automation available on high-end machines such as use of GPS to control cut/fill operations with bulldozers, and laser-transit-based automated dig depth control for excavators.
[1] https://www.cat.com/en_US/articles/for-owners/the-basics-of-...
If you're going to automate the kinematics, then you either need a manual override mode (which the operators will be unskilled with since it's seldom-used), or you need sensors and rules for the machine to know where it's not supposed to be (clearances around wires, other structures, etc) and that needs to be better than human operators currently do.
Those are theoretically possible, sure, but a valve-per-cylinder is really stinkin' reliable.