"I simply cannot understand how people can say it's either immoral or unethical to walk away from a house"

You signed a contract. That's what makes it unethical to walk away. You put yourself in the position that you are in. You may think you took a better job offer, but actually if you had considered your circumstances, the job offer was not that great because of the obligation to the mortgage company where you lived for the house that you bought.

You say that you could "barely" afford the mortgage payments which implies that you could afford them. The last paragraph of your response is just a list of excuses to make yourself feel better about the course of action that you have taken. Your part of the problem, not the solution.