I lived there for 12 years. Generally speaking, Baltimore is good if you want relatively affordable housing. A number of neighborhoods are great for DINKS (dual-income, no kids). Public schooling is a hard no-go unless you really get involved and enroll your kids in a carefully selected charter or magnet school. There are some excellent private schools. Baltimore county has options too if you want something that's more suburban.
As far a crime goes, the city has some really profound problems that are the result of white flight and systematic disinvestment. It was hit hard by the crack epidemic and has not really recovered. The TV shows "The Wire" and especially "The Corner" capture the flavor of the rough parts of Baltimore more than residents would like to admit. As far as safety is concerned, like other big cities, it's not really a problem for folks that would read HN, unless you enjoy hanging out in bars at 2am and getting into arguments with armed drug-trade people. Don't leave anything in your car, get a security system for your house.