IANAD but that sounds serious to me. Visual field distortions are often brain issues. I would go to a neurologist to be sure. Could be a tumor applying pressure to your optic nerve or something.
Anyway, visual field deficits can be brain-lesion-driven, as you say (e.g., the classic bitemporal hemianopsia in pituitary tumors).
In their case the doctors didn't remove it because the operation would have been more dangerous than the tumor itself, and the symptoms were not too severe. But it's obviously better to be aware of such things.
ElongatedTowel ought to go to the doctor and deal with a few weeks of feeling like an embarrassed jerk by demanding that they scan his/her brain.