It's mostly a cost issue. Towns see the huge maintenance bill and decide to close their pool. Half the indoor pools I went to in my childhood are now closed. No new public pools have been built.
When I was a kid, I had swimming classes every other week in school. My kids have swimming classes 4 times a year because the few remaining pools that schools can go to are overbooked.
The completely predictable result is that the number of kids and adults who can't swim is growing every year.