People want to tell stories about the past, but they completely ignore the more recent past, specifically, Generation X, the notorious "latchkey kids", who also tended to belong to very small families, because the entire generation was the smallest in population of the recent generations.
US birth rates dropped dramatically between the late 1950s and the late 1970s, resulting in this small generation. (I attribute the drop to the increasing financial independence of women and the widespread adoption of birth control.) GenX family sizes were small, with few siblings. Yet GenX kids "enjoyed" (or suffered from, depending on your perspective) the absence of supervision for large parts of their days. Helicopter parenting didn't exist at that time. GenX is the glaring counterexample to the family size theory.
The past with extended families all living together — grandparents, cousins, et al. — is more distant than people are acknowledging.