Even if you want to differentiate "who" and "whom", "Whom were you speaking with?" is more natural.