i had a ton of fun during a few of my MIT P.E. classes and ended up incidentally meeting at least one guy whom i became good friends with later in life (when people ask where we met, "P.E. class" is an unexpected answer).

there might be some value in "forced" non-academic activities, as long as people don't take it too seriously. iirc the P.E. teachers were often grad students who loved their sport, not jaded old fogies from middle school nightmares.