What happened is that, historically, Baltimore-Washington was mainly served by the B&O, whose Baltimore station would have been right next to Inner Harbor (Camden Yards is essentially built on old railroad property). When passenger service was consolidated into Amtrak, the Northeast Corridor instead chose to use Pennsylvania Railroad trackage instead of B&O from DC to NYC, and the freight trackage in the corridor consolidated onto the B&O tracks instead. The resulting residual commuter rail line has the standard commuter rail/freight rail politics going on that limits the number of trains that can transit the route.