Giving out mortgages is not difficult. You just have to be a bank. Not only that but 2007 showed that profit-making banks are even worse than a reasonable person, or precocious five year old, would be at deciding who should be given a mortgage.
Mortgages are a necessary part of society. Profit-making banks are not.
The only reason you spend your life slaving away paying extra interest to the shareholders of banks, is because banks have manoeuvred themselves into a privileged position. Inequality is not an accident of the system, it is the point of the system.