>If a developer is given free reign as to what they can build, they will naturally build the most profitable thing they can on the land they have available. I doubt anyone would dispute that?
Nobody disputes this, it's obvious. However, you're missing the benefits of this approach though. For a lot of industries you have very high upfront costs. You want to pay those off as soon as possible. Once they are paid for you can often drop your price substantially. The company has to pay for the training of the construction workers, lease machines and pay for the currently very expensive housing because that is where the construction workers live.
>The idea here is that the most profitable thing tends to be more upmarket than the locals can typically afford. The area gets gentrified and rents go up.
Well, this is where things go wrong. You are mixing up cause and effect. Demand (gentrification) leads supply (house construction). If you fail to construct new housing you get displacement. That demand is driven by people getting a well paying job and they have to live somewhere. If you don't build housing for them they will have to displace locals because that's the only place where they can live. From this perspective the new comers don't want to take your home. You can keep it. They just want a place for themselves.
>A socialist would say that if the market fails to provide solutions for core needs of people (health, housing, etc) it is up to the government to provide those. Some things are simply either unprofitable or not suited for a profit driven approach (e.g. private prisons where the companies running them are actively incentivised to see repeat offenders).
That's not a failure of the market but a failure of policy. The primary reason why public housing works is because it forces the government to acknowledge bad policies and replace them. The bigger failure of for profit housing is that it is slow. A housing shortage that took a decade to create also takes a decade to solve. The government has the benefit of being excellent in the short term but in the long term it is only "good enough", which is why you need both.