People weight "career downsides" differently. A lot of folks who are further in their careers are done trying to climb the ladder and would be happy working remotely.
Two other cynical additions. Not everyone is as concerned about "productivity". Also, there's a nonzero chance that someone whose productivity drops dramatically when working from home were only ever productive because they were leeching productivity out of others. Every team seems to have someone like this, and they can be safely gotten rid of.