I think the biggest problem, the obvious one, is crime. There are some small pockets of intense gentrification, and even these areas aren't particularly safe. The rest of the city is a no-man's land. Literally within a few blocks of crowded, active areas (e.g. Fells Point) there are extremely unsafe blocks where the chances of a mugging (or worse) are unreasonably high. Then, there are huge contiguous swaths of the city where no one but residents are really advised to enter. I remember going out to a bar in West Baltimore and not a single cab was willing to drive into the neighborhood to pick us up. The patrons of the bar refused to let us walk home ("you're gonna get stabbed!"), and a kind stranger drove us home.
Most people I know who live in Baltimore want out, and many either move to DC or to the suburbs.