Depends, if it's tensor parallel or pipeline parallel. Only PP doesn't pass too much. TP does