(Cosmologists like to use particular systems of coordinates which are comoving with the expansion to capture the fact that no forces push these clusters off their inertial motion. However, there are arbitrarily many systems of coordinates which are not comoving with the expansion, and switching to any of those means that your coordinate distance to faraway clusters of galaxies isn't constant. In some of those systems of coordinates it is easy to be misled into seeing https://en.wikipedia.org/wiki/Fictitious_force . Again, a change of systems of coordinates, especially to comoving ones, can make those frame-dependent forces vanish.)
One can also uses systems of coordinates to treat the metric expansion of space as the increase over time of a gravitational potential measured outside a galaxy cluster. The metric expansion means that over time an object has further to fall from outside a galaxy cluster to one of the supermassive black holes inside the cluster, with related observables like an increase in the gravitational redshift of light climbing out of the cluster.
The same extra-galactic observer sees an increase in its gravitational potential relative to every galaxy cluster in the universe, because the radial distance (in spherical coordinates) to all those galaxy clusters increases over time.
We can substitute an entire galaxy cluster for our isolated human-like observer without substantially altering the picture I'm trying to paint above.
If there is vacuum everywhere:
vacuum --- us --- vacuum --- galaxy --- vacuum
Why would galaxy go away from us? I expect that the vacuum on the left of the galaxy applies the same force on it as the vacuum on its right and thus should have a null net effect on its movement?
Using your same diagram
vacuum <---- us -----> <----vacuum----> <-----galaxy---->