It's because I think that this design is the most efficient use of space, if you want to ensure that Game Boy cartridges don't stick out (like they do on the GBA). The device is divided into a front half and a back half, with the PCB in the middle. The screen and buttons are all in the front half, and the cartridge and battery are in the back half. If the device were landscape, you'd have the cartridge take up the entire middle space in the back half, leaving smaller spaces to the left and right (under the Dpad and face buttons) for the battery. So you'd either need to use two smaller batteries (and deal with that), or use a single battery (and have worse battery life and an empty space).
It's definitely possible, this just seemed like the simpler way forward. I probably would actually personally prefer a landscape orientation. Maybe in a future version!