Mostly the confidence comes from half a century of trying to find such "extra" visible mass! There's a long history of people looking for extra dust or gas or compact objects using a tremendous variety of clever techniques, and yet the deficit has remained. At this point, if you believe that there indeed is extra mass around, it kind of has to be dark matter by definition because it's been so damn hard to see.