Here's a good explanation I found quickly via Google, I'm sure there are others: https://medium.com/mydex/we-can-afford-what-we-can-actually-...
No evidence is mentioned that there was enough available in the way of bricks, mortar, steel, cement or labour. The banker quite likely hadn't checked any statistics or made any estimates. There is no particular reason to consider his opinion even informed on the topic. Britain had just had suffered around a half-million war casualties drawn disproportionately from their able-bodied labourers and their factories for things like steel and cement had probably suffered a few hard blows and were likely being run ragged.
They could have forcefully reallocated resources away from something people thought was more important towards building houses; but the issue that is being glossed over is if someone thought that there were higher priorities than building houses ... what if they were right? The thing about economics is prices aren't arbitrary, they encode a combination of how easy a good is to produce and how much demand there is for a good. They can't just be overridden expecting a good result without factoring in those things.
If Britain had some magic ability to just build whatever they want and there were enough resources available to do whatever then the empire wouldn't have collapsed shortly afterwards. They were way over-extended in terms of what resources they needed vs what they had available.
Your second criticism performs the standard hijack. The natural follow on is "if we can do it, and we can afford it, should we do it"? Should we do it is the right question to ask. Just because we can and we can afford it, doesn't mean we should do it. But far too many people try to squelch the debate by asserting that we can't afford it. We can afford universal health care or a modest UBI or a green new deal or whatever expensive thing that the left wants. Whether we should is the question, not whether we can.
"You can afford what you can do" is a truism - if you can do something, the financial system will tell you that you can afford it except if something really weird is going on. If the resources were there to build the houses then the prices would shuffle around until it was also affordable.