20,000 may be small but for those of us who grew up in rural areas I don't think that's what we think of immediately when you invoke the phrase "small town." For us the 20k town was the booming city we would sometimes visit.
From where I grew up in the country we could drive about 20 minutes to get to a town of maybe 1000 people and it's very different than 20k. You probably have one small clinic in the 1k town (compare to multiple doctors and probably a hospital in the 20k town) and maybe a small independent grocery store (vs probably multiple regional chains in the 20k).
The difference of scale is a difference of kind too. And also, the image of the "small town" - the romanticized version - isn't of the strip mall laden 20k town, but of the 1k town frozen in the past.
But let me tell you both of those size towns suck for different reasons. The decay is more palpable now in the 1k town as the economic horizon has really contracted. That one grocery store is probably closed now. The 20k town may make a go at things because it has more residents and thus retail type jobs, but unlike the 1k town built environment is more gross and less quaint, and the strip mall filled landscape is depressing.