"Giving up on housing" is not the right answer. "Start small and work your way up", like most of us did.
See, when you hit 20, you're looking at your parents, and you're seeing them when they're 45 or 50. You're not seeing them when they were 20. You expect that you're going to be able to live like they do at 50, not like they did when they were 20. That's not realistic.
But the answer isn't to give up. The answer is to keep working on it, over decades, instead of expecting it to be handed to you on day one.