There are 4 segments and which one is being used is presented on the bus interface (S4/S3) so 4MB of total memory is actually possible in an 8086 system, and the 8086 documentation does explicitly mention that as an implementation choice, but I don't think many systems made use of that feature; bank-switching seems to have been more common.

The reason being that separating the segments into their own address spaces would create something more like a Harvard architecture, which isn't really wanted in a general-purpose computer.