Define "holistic understanding."
Is the doctor actually sitting bedside with the patient for the entire shift? Of course not. More realistically, the doctor is coming in, making some observations, and then moves on to the next patient.
So here's an idea - couldn't you shoot video of the observation, and shoot a time-lapse video of the otherwise-lacking-direct-observation periods, and show those to the physician in the next shift? All that the attending physician has to do is vocalize his observations (of course a camera by itself can't sufficiently observe), and the hand-off physician can watch the clips just like he could read the chart. Such clips may also be accompanied by speech-to-text to produce quick summaries.
There's simply no doctor in the world who isn't privately treating a single patient who can compete with the number of experiential data-points which a computer can provide.