I think the problem is that you are thinking of things purely in terms of how gas stations and gas powered cars work. You need to step back and re-think.

For almost all EV owners, a level 2 (240V 20amp) charger at home is all that's needed[0]. It charges slowly, but given the car sits for 12+ hours every night, it's more than enough. You get home, you plug in, you go eat dinner and go to bed. When it's time to go in the morning, you have a full charge. "Refilling" isn't a discrete event that takes place in a given place. You just plug in whenever you aren't driving.

DC fast charging, which charges the car in an hour or less, that's a rare event. It's what you use when you're on a long trip somewhere that takes more than one charge of the battery. But most owners might do that a few times per year.

So the problem you're trying to solve with this "swappable battery" solution is tiny. Imagine how much more the car will cost to support this complexity, and for what benefit? Saving a few hours per year? I won't pay the additional thousands of dollars for the "swappable battery" car, and I won't pay the cost of maintaining the swappable battery stations. It's just not worth it vs spending that time getting lunch.

[0] My wife and I did the math on our own driving patterns and realized we could probably get away with a level ONE charger- literally a standard 120V extension cord - for the EV we're on a waiting list for.