It's not that I don't see them other places, but it was a lot more concentration there and then... and that was memorable.
If you are a tourist, you're likely to be in the more aesthetically pleasing areas or commercial areas and during the work week.
The counter example would be if you were in a small town an hour beyond the outskirts of the larger cities and went to a local restaurant for dinner - the age demographics would be quite different.
One of my taxi drivers look so elderly I dared not let him even touch any of my bags …
Of course, this isn't really that unusual in developed nations, but it's more pronounced here.
On the plus side, Tokyo is a really great place to raise a family unlike American cities, because it's safe and walkable. Your kids can ride a bike, or take a subway, by themselves if you want, and not have to worry much about safety.
Btw, are you a new parent? That tends to color things imo, you notice a lot more babies than otherwise.