The universe is young, a lot of stuff hasn't happened yet. It wouldn't be surprising if we are about the earliest life of this sort that could exist.
At least, I can understand visualize the time scale of the universe a lot better than I can visual the length scale. Speed of light doesn't feel like 1/1 to humans!
Alternatively, map all the white dwarves, then wait long enough, then look again..
A fully black dwarf would be so cold it would be indistinguishable from cosmic background radiation, but that's going to take at least a trillion times longer than the current age of the universe. Not just slightly longer.
A WD will slowly cool to a BD over Trillions of years, so it's all linear. Funnily enough, they are probably every shade of red/brown in-between - but "brown dwarves" are usually reserved for objects that never where "proper" hydrogen-fusing stars, but are large enough to fuse deuterium.
Possibly through gravitational interactions with non-dwarf stars in the same system.
There not being any yet makes either method somewhat more challenging.