I assume the center of gravity of such a world would be the actual center of mass of however the voxels are arranged, meaning players could affect it with large enough structures or small enough worlds. Without adjusting "straight" "horizontal" structures to conform the curvature of the world, gravity would no longer be a force that was normal to the surface of the structure at some points on it.
I can also imagine people digging to the center of a world to find the gravity flip-flop point and playing with it. Or hollowing out the interior of the world in order to get a zero-G chamber (assuming no atmosphere) [1]. Or building gravity trains [2].
[0] https://imgur.com/a/0J5RoE1
[1] http://hyperphysics.phy-astr.gsu.edu/hbase/Mechanics/sphshel...