Now, regarding social time with people I'm not doing projects with, I'm all for that, since information does not always flow freely within a company, and talking to people on adjacent projects could lead to new knowledge or collaborations, or at least be set up for such. It's also good if you become a hub of information for other teams, as that increases your value to yours and the teams you talk to. Information is power :)
Of course, as you said, you have to weigh this against other skills you can spend time on, it all depends on what you want from your career. If you desire to move up the hierarchy, soft skills such as this will grow in importance, as your work shifts from being neck-deep in technical details, to negotiations with other teams on how to achieve large initiatives. It's easier to catch someone's ear to talk about something you'd like to get done when they have a positive social impression of you.