Or what if Dark Matter was lots of Black Holes? We (more or less) know what Black Holes are so why don't we assume there could be lots of them having a gravitational effect but no visibility?
In other words why do we need to assume that Dark Matter is made of some kind of exotic unknown matter?
Black holes are indeed[1] a dark matter contender!
Just because the best explanation[2] we have right now is cold dark matter (CDM), doesn't mean physicists are content. They're looking for new ways[3] to probe not just CDM but other dark matter models as well, like primordial black holes[4].
Dark matter is by no means a settled topic.
[1]: https://en.wikipedia.org/wiki/Primordial_black_hole
[2]: http://pirsa.org/20040087/ "Dark Matter: A Cosmological Perspective", nice colloquium talk giving overview of what we know about dark matter.
[3]: http://pirsa.org/20100052/ "Strong gravitational lensing as a cosmological probe: new ways of constraining dark", random technical talk about new way of probing CDM.
[4]: https://arxiv.org/abs/2002.12778 "Constraints on Primordial Black Holes".
"What If (Tiny) Black Holes Are Everywhere?" https://www.youtube.com/watch?v=srVKjWn26AQ
Dark Matter is just a placeholder name, don't read too much into the "Matter" part. There are no assumptions there and plenty of alternate explanations are proposed, like Modified Gravity (MOND)