A) Consider the technological acumen of the average filling station worker. Is that the guy you want to be in charge of determining whether any given battery is fit for reuse?
B) Consider the weight of an average automobile's starter battery. Not the sort of thing that a petite woman like my wife would be able to sling around on her own. If you hire battery jockeys to swap them in and out, that increases the cost.
C) Now consider the fact that an electric car requires a LOT larger battery than a starter battery. After all, the starter battery only has to turn the starter motor for a few seconds; the EV battery has to turn the entire drivetrain for the entire time you're traveling between charging stations. If you broke the battery modules up into easily manageable pieces, it would take just as long to pull them all out and replace them with fresh ones as it would to just charge them up in place.
D) Consider that because an EV battery is so large, they tend to be located in fairly inaccessible places--under the passenger compartment, molded into the frame... Thus, it is not a simple job to remove and replace.
There are probably other obstacles to implementing a battery swap scheme that I'm not seeing off the top of my head. And none of these are truly insurmountable, but the solutions tend to involve the movement of pieces of green paper, or the big hand on the watch dial.