Thanks, I didn't know what they were called!
There must be some granularity at which voxel engines can work smoothly today. Let's say my room is a square with 320 pixels on each side. 320^3 = 32,768,000 voxels, which looks like a lot, but an RTX 2080 can render 17 billion triangles per second, which works out to 273 million per frame at 60fps.
Let's ignore gravity and focus on momentum vectors and collisions. Even if simulating physics for each voxel were 10x slower than rendering a triangle, you could do 320^3 voxels in 60fps on today's hardware.
And this is naive, assuming you can't merge neighboring voxels together! I think you're being unfairly pessimistic.