On the other hand, simulating say skin/muscle/blood or anything that closely resembles human body is near impossible. Without that, a simulator is pretty much useless and it'll probably easier to train the surgeons on real robot + some animal like pig
To run a high-fidelity simulator of human body that is useful for surgeons, you need a LOT more. And I doubt it can be data-driven. Data-driven simulators for things like autonomous cars are just coming up, and these are way simple as the agent (the autonomous car) doesn't directly change the environment when it's driving around (as in, you don't have to simulate the car colliding into a traffic pole and then breaking it, etc.)
To simulate a human body, you need to be able to capture the material properties of different layers, some of which are fluids, and then also the interaction between them and how different organs react to a surgical operation. It's a very hard problem.
Why do all that when you have animal corpses readily available? There's no need to create a problem that doesn't exist.
a) thousands of hours is far far far too little data. This is a total ballpark estimate, with only knowledge of the ML and basically zero knowledge of the surgery side, but I would expect three to six orders of magnitude greater, i.e. millions to billions of hours necessary to train a machine learning model to do something like that, to a standard.
b) the big problem is not the procedures themselves, but edge cases and things going a little bit wrong.
c) you are fine with tiny flukes in your generated images, but you won't be very happy with an ML model whose tiny flukes lead to internal bleeding.
d) unfortunately (with contemporary models) we can't easily explore latent space, which might possibly contain regions corresponding to catastrophic robot arm movements. the chance isn't that high, but it's not a chance most'd be willing to take at this point.
In the case of robotic surgery simulator, you are using the controls to control the arm, but to interact with what? If you just want to move the arm around and maybe interact with some rigid objects sure that's easy. Would that add any training value to the surgeon? Probably not. You can get value only when the simulator includes a simulation of something the surgeon would have to face eventually - organic mass of the human body. Simulating that is hard, and I doubt anyone would invest much into it when you can train surgeons on alternative physical objects like pigs.
I'm working for a small company that is trying to bridge the gap and make something good that can run on the consumer VR/AR hardware that's coming out in the next year or three. Lots of interesting problems to solve.
https://www.intuitive.com/en-us/products-and-services/da-vin...
In my experience in a related field, simulators are about 2% as useful as advertised. The cousin thread explains how difficult building a flesh simulator is, so I wouldn't expect surgical experience on a simulator to be very useful.
Surgeons can take a few practice simulations in the Sim, but it's just that doing so ties up the robot from doing real cases at that time also.
So, pretty cheap, it sounds like? How much is the hourly rate of surgeons?
A surgeon makes on average $115 an hour in California, $280k a year. So the console plus simulator backpack costs a surgeon or two's yearly salary.
https://www.ziprecruiter.com/Salaries/Surgeon-Salary--in-Cal....
An additional aspect of simulation is calibration and use of phantoms [1]. These are materials of known characteristics approximating human anatomical densities. I suppose for robotic surgery this would be used for both the imaging and the surgical tech.
0. https://radiationoncology.weillcornell.org/clinical-services...
This same limitation applies to simulators though, so this application isn't an answer to the parent comments question.
Here is one example paper: https://pubmed.ncbi.nlm.nih.gov/29023350/
The author wrote: "The paper I published in 2019 summarized my findings, which were dismaying. The small subset of trainees who succeeded in learning the skills of robotic surgery did so for one of three reasons: They specialized in robotics at the expense of everything else, they spent any spare minutes doing simulator programs and watching YouTube videos, or they ended up in situations where they performed surgeries with little supervision, struggling with procedures that were at the edge of their capabilities. I call all these practices “shadow learning,” as they all bucked the norms of medical education to some extent. I’ll explain each tactic in more detail.
"Residents who engaged in “premature specialization” would begin, often in medical school and sometimes earlier, to give short shrift to other subjects or their personal lives so they could get robotics experience. Often, they sought out research projects or found mentors who would give them access. Losing out on generalist education about medicine or surgery may have repercussions for trainees. Most obviously, there are situations where surgeons must turn off the robots and open up the patient for a hands-on approach. [...] My data strongly suggest that residents who prematurely specialize in robotics will not be adequately prepared to handle such situations."
The author also listed examples of accessible simulators, notably one that uses virtual reality: "In the past five years, there has been an explosion of apps and programs that enable digital rehearsal for surgical training (including both robotic techniques and others). Some, like Level EX and Orthobullets, offer quick games to learn anatomy or basic surgical moves. Others take an immersive approach, leveraging recent developments in virtual reality like the Oculus headset. One such VR system is Osso VR, which offers a curriculum of clinically accurate procedures that a trainee can practice in any location with a headset and Wi-Fi."