It would also cause a lot of confusion, where clueless older relatives would buy games for kids, not realize that an accessory was required (or have no idea if the kid actually had that accessory), and then the game wouldn't run.
We see this sort of problem happen a lot with computers of the 8-bit era as well, where add-on modules would fix a lot of the issues with the base system... then be supported by almost no software for these exact reasons.