Pressurizing has weight advantages. You can use a balloon tank. The tank no longer has to have the rigidity to support itself.
Rockets with typical cycles have used balloon tanks. Even modern ones like the Falcon 9 are partially that way, with just enough rigidity to be erected empty on the pad. The Falcon 9 uses helium to pressurize; that could be changed to preburner exhaust even if you did keep the moderate pressure and the turbopumps.