I figure at least some of it comes from the idea that mathematically, a singularity is a point (e.g., in the graph of z=1/w, there is a singularity at the point w=0, and in the graph of z=(1-w)²/(1-w) there is a removable singularity at w=1 (that is, the function is undefined at w=1, but if you put a point at (1,0), the graph will be continuous and no longer have any holes in it). The fact that both have the same name and the similar behavior of a black hole singularity to a mathematical singularity¹ can lead people to make an incorrect assumption.
⸻
1. I must admit to a lack of sufficient GR education to feel confident in this, but I think that one of the issues that made physicists unwilling to accept the idea of black holes when they were first postulated was that there ended up being a division by zero in the mathematics.