But the benefit of virtual memory is the contiguous memory (that those virtual indirections allows for) - ignoring virtual memory we risk falling down again to the "you need 600kb [of contiguous] conventional memory" issues of fragmented memory.
A MMU is complex, but allows simpler programs in the entire system overall.