Then there's the legal minefield. Being a developer or landlord makes one person very rich very quickly, and that person is your lawyer. Especially in the Bay Area. People here pretty much threaten lawsuits if they made eye contact with a teenager on the sidewalk, or if their cat's not eating its food, or if they don't care for the weather on any given day.
Source: I own a small apartment building in LA and sit on my HOA board in SF. I am the definition of a small-time real estate owner in the uncanny valley: not poor enough to not give a shit, but not nearly rich enough to do so, either.
I'm not a full-time landlord; it's more of a nights-and-weekends thing. But it sucks up a not-very-fun proportion of my nights and weekends, and every now and then, it eats into my workday as well. I would recommend landlording to people born into significant wealth, who can buy multiple properties and let a management company deal with the headaches. I would not recommend it as an upward path for those who aren't prepared to make real estate a full-time job, and who can't afford to jump a few rungs up the ladder at the outset.