But there are lots of trains leaving at any given time going to different places. There's no way to know which train somebody is heading to without asking them.
As somebody who commuted for a long time at peak rush hour back and forth from Penn Station to D.C., I can tell you that following people who "seem to know where they're going" would not be a reliable strategy.