But with only four segment registers, you couldn't treat it as that - not unless you only had four objects. So treating it as a sliding window was all you actually could do with it.
Using a bare segment register as a pointer was quite common. That’s what the DOS memory allocation call would return.
In retrospect, they seem so useful and obvious but apparently not. I guess the type of person who thinks near/far/huge is extremely complicated found segment pointers even more complicated.