Seems like a reasonable default would ask for your second factor if you attempted to remove it or add another, as these are potentially account compromising events. And since these changes should very rarely happen, it's not exactly an onerous requirement for everyday users.