“With whom were you speaking” IS grammatically correct though. “Whom” is the object of the preposition “with”.