People aren’t less fertile (or maybe they are - obesity etc) - the additional factor is that they’re choosing to not exercise their fertility.
https://data.oecd.org/pop/fertility-rates.htm#:~:text=Fertil....
If people can't have children you can't fix that as easily as the factors causing people to not choose to have children.
A change to tax codes can immediately help alleviate the latter, for example.
Of course you are right that the underlying reasons are all relevant depending on the question, but they are all covered under the umbrella of "fertility rate".
In conclusion I am (pedantically?) arguing the definition of the term Fertility Rate, not anything else.