In the last model they trained they shared some part of a layer for all the agents. So it was like a one agent trained with shared knowledge of the rest.