I agree with your point, the bar IBM was shooting for was set by existing popular microcomputers circa 1979. The only significant consideration for future growth/competition was seemingly that the established trend of RAM size growth would probably continue. At the time there wasn't really any established trend of progressive growth in graphics resolution or colors. Pre-Apple II examples like the Cromemco Dazzler for the Altair weren't fundamentally different than the Apple II and probably not even on their radar due to being barely out of the kit/hobbyist level.
I'll add that when considering the 5150's initial design, the "IBM" we're talking about isn't really "The IBM" but rather a sole skunkworks project located in a backwater division down in Boca Raton Florida intended as an experiment to learn more about these new microcomputers. Most of the rest of the traditional IBM management structure barely knew about it during development and those parts that did mostly ignored it. If 'mainstream IBM' had approached the PC as a real IBM project, it would have certainly been very different and probably unsuccessful (if it had managed to ship at all). As it was, the 5150 was only able to use off the shelf components (including the CPU) because it was considered a one-off experiment initially given a month for the design and a year to ship.