Well, if it's the same git server using the same SSH key for every repo, can't you still use SSH's ControlMaster to open the connection once and leave it open?