But maybe the stock of primordial black holes started as Dark Matter particles. They then have the problem of explaining why so much of the Dark Matter started out in or close to a black hole. Maybe each Dark Matter particle is so dense it is a black hole all by itself?
You have to be careful talking about what a black hole is made of. Stuff that falls in only has a description before it falls in. After it falls in, it can't be thought of as made out of anything, anymore. So if black holes were part of the initial universe, there is no saying which flavor of matter they represent. After things have settled down, these black holes could proceed to swallow any amount of regular matter, and any amount of "other stuff", if there is any.
The assumption is generally that the "other stuff" has difficulty cooling off enough to be sucked wholesale into black holes, unlike regular matter that is happy to radiate away a big fraction of its energy as photons, leaving the rest able to collapse and form planets and stars, and thence perhaps be gathered into the nearest black hole. Meanwhile, the Dark Matter can mill around a black hole's vicinity indefinitely, maybe swooping close and getting ejected, but otherwise hardly bothered unless it blunders straight in.