Science is prodigy but not magic, so when we try to explain things that are way too complex or out-of-scale (whether big or small) relatively to us, the solution space is so incredibly vast that we might as well be guessing — science lets us rule out 1% of the 'whys' and we're almost none the wiser.
Consider this:
- galaxies may generally follow cycles, patterns, notably depending on the activity of their supermassive blackhole (or quasar, etc); life may be peaking during a "sweet spot" in the cycle, and plummeting at other times. What may be critical is thus not only age but 'state' of the galaxy, and we might find ourselves in the beginning of such a cycle (we sure accumulate enough particularities in the topology of our local spacetime, being both rich yet calm enough, to warrant the assumption that we're currently 'privileged' on average).
- We have no way to detect life in other galaxies. They could be swarmed by spaceships and interstellar spatian empires and we just wouldn't be able to determine that. Thus basing the Fermi "paradox" on such a small sample (on the order of 10⁻¹²) seems much too weak data. We simply don't know. We actually can't properly observe ours yet, especially behind the core, generally in the alignment of the disc. We are just now discovering that said disc is distorted and there are remnants of jets above/below us[1]. There are hundreds of galaxies in our local Andromeda-Triangulum region, huge gravity streams, and we've only begun to grasp that in the last few decades... It's still very, very early in our observations of the cosmos.
I think it's good to ponder such questions and raise apparent "paradoxes", go science! — but to draw any conclusions based on weak data and conjecture of a perceived (read: biased) mean is, IMHO, rather foolish, way too premature at best. Not hubris but wishful thinking, belief, opinion. So let it be that, humbly, defusing the anxiety.
In the grander scheme of things, we're not much better off than the Ancient Greeks when faced with this question. Give it another thousand years, and then I'd expect a whole other level of questioning. Not 10 or 20 or even 100, if we are to tackle that kind of scale — we need infrastructure for this, in space, way beyond the solar system (think telescopes, probes, not even manned missions).
The ultimate problem with time is not evolution (of us, of science, of tech), it's information travel: e.g. if we were to use the thickness of the Milky Way to set up telescopes, at ~1,000 LY, it would take half that time to transmit pictures to a central aggregator. 5 centuries for your next picture, I don't know if we'll be able to commit to such transgenerational projects without huge changes in our biology (lifespan, mindset, whatever but huge, enough to make us a different species).
Until we're able to do such things, grow such infrastructure, gather such meaningful data, we're just too small to even look at the cosmos. Let alone draw conclusions. We really just have to keep going at it, and it's gonna take a long, long time.
[1] Contrary to the very regular spiral illustration we're acquainted to, we are now realizing the Milky Way could actually look more like this: https://en.wikipedia.org/wiki/Centaurus_A