The single biggest reason is that it locks you in, and unless it becomes a standard it's basically useless since people would probably need to go to a dealer to get a battery swap. For the average driver of a BEV, they are rarely going to need a fast charger anyway since just charging at home overnight is good enough for 95% of driving.

That's less of a problem for fleet operators, they could actually have a bunch of batteries at their business sites spread all over a city so a battery swap is never too far away. Fleet vehicles also do more km in a day than the average private vehicle as well, so they WOULD need to use fast chargers a lot more often.