The most likely answer (and one we don't want to admit to ourselves) is that intelligent species have a certain lifetime limit. Most species die off within 3 million years.
An alternative answer would be that advanced intelligence figures out a way to delve into mind or other dimensions, rather than outwards into physical space.
- Intelligent species are extremely uncommon - life could be common, but mostly bacteria / simple single-cell.
- Intelligent species choose to hide themselves from us or others somehow - I'm no cosmologist, but maybe some of the dark matter we've calculated be accounted for by dyson spheres?
- Intelligent design - not personally a proponent of this approach but any form of intelligent design (fundamentalist/whole-earth creationist or adapted to evolution) could explain the dark sky paradox
Required (fiction) reading on the implications of Fermi Paradox:
- Mass Effect
- Stephen Baxer - Manifold books, esp. Space and Time
- Too much Asimov to list
Looking at our own biochemistry, you don't just need carbon, nitrogen, oxygen, and hydrogen to make it work. Heavier elements from the fourth and fifth periods of the periodic table are also required.
I wouldn't be surprised if life is rare but also becoming more common over the last few billion years. Evolution also doesn't necessarily lead to intelligent life capable of harnessing technology.
The dolphin will never make a spacecraft, even if he is twice as smart as we are. Lack of hands is a problem. Also, lack of ability to create fire is a problem. Also, lack of ability to easily pour liquids from one isolated vessel to another, without diluting them is a problem. Basically, no "water worlds" will create a technological civilization, in my opinion.
There's also something cultural that may not happen for some reason. Look at the Native Americans. They still didn't have the wheel or metal, four thousand years after others had them. If there is nothing forcing you to develop tech, you won't, even if you're technically smart enough and have the resources that would allow it.
I can also imagine some sort of plateau being reached, after a particularly nasty government takes over. What would happen here, if certain countries/groups were suddenly in control of the entire world?
We may already be reaching a plateau ourselves. The desire is waning, I think. It has certainly waned in my lifetime, both personally, and as a world, I think. At some point, you become "practical", and decide to expend your time and resources on things closer to home.
Add it all up, and it's at least a couple more multipliers for your fermi equation.
Maybe by the time lifeforms get the technology to travel to other stars, they find that they don't need to expand.
There are many places (even within the USA) that we have the technology to colonize, but not the desire.