Housing hasn’t been the biggest problem in Japan since 2000s. Like it is, but not the biggest one. Yet the birth rates have never increased after the initial bump.
People, especially women, have other things they can spend time on instead of giving birth to 3+ kids.