So you said "use yum/apt to ensure software packages are up to date, or at a certain version."
So how do you make a 100 node cluster get to the desired installed state? Do it by hand? Ssh loop? cssh or similar tools that attempt to do parallel ssh? This is the part where things really break down and then I realize there actually is no principled way to do it, and in fact it's done by hand, and the sysadmins are just creating more work for themselves ensuring job security.