I found one of the comments very thought provoking: "The declining total fertility rates are children not born in the moment, but the hope is that they are delayed, not forgone".
I hadn't considered it before, but a gradual change in what age people choose to have children would show up as a temporary drop in fertility (for an easy example, think about everyone delaying their next child for five years). A quick google search show's a strong correlation between steeply increasing age of first child and low apparent birth rates. In particular, US fertility rates were even lower in the 1970's, another time when age of first child was increasing rapidly.
Back of the napkin math shows that we'd see (very roughly) 80% of true fertility if not adjusted for, making the true fertility rate slightly above replacement levels (about 2.2)