The real problem as always is zoning. Houses cost roughly the same on a $/sqft basis, adjusted for inflation, as they did in the 1970s. They're more expensive now because outside of town they're twice as big on average [1] - in part due to zoning rules. In town, zoning rules preclude densification necessary for supply to meet demand [2]. And, families are smaller.
Check out Japan housing CPI - a market where supply meets demand thanks to federal zoning rules, and is also centrally banked. Dead ass flat. [3] This chart alone basically debunks the theory the Fed is solely responsible for lower housing affordability.
Zoning is the issue, not the Fed.
[edit] Yes, the fed contributes a little - monthly affordability of a more expensive house is the same when interest rates are lower, but down payments are not. However, if you want to solve the affordability crisis build more houses. Any houses. Now. Compare the Japan housing CPI in [3] to the US housing CPI in [4] and weep. Housing in the US is driving inflation, not responding to it.
[1] https://fee.org/articles/new-homes-today-have-twice-the-squa...
[2] https://en.wikipedia.org/wiki/San_Francisco_housing_shortage