This is very possible.
Do it when you are young though. Don't wait.
Start learning now.
If serious, and have the time, land, and funds for materials, go out and buy pretty much every book related to building Taunton Publising offers.
Start with Working Along. A Taunton book with tips on Working Alone.
Learn Electrical Residential wiring early on. (I have found most DIY'er have a terrible understanding of Electrical Theory, Layout of branch, home-run circuits, and all the NEC codes. It's not that difficult.
Learn about Plumbing/HVAC. You will be suprised it's not that difficult. (Radiant heating is not that complicated.)
My biggest regret is not building a house completely in my thirties. I was basically a Contractor Handyman, and never build a house from scratch.
And to be honest, even though I have accumulated the knowledge to do the job now; I'm just not physically what I was in my twenties, plus I'm filled with neurotic anxiety.
It's ironic, when I was in my twenties/thirties, I didn't have all the knowledge to build a large house. Now I have the knowledge, I question my physical abilities. And I'm scared of risk more than ever. In so many ways, I have given up. If I had a kid, our first project would be building him/her a house.
That's why I say do it young. What makes it really feasible these days are the quality of the info out there.
To the poor kids out there--really try to build that house early on. You won't regret it. It's better than any stock portfolio.