The best advice I can give is to use the Jenkins plugins wherever possible. Jenkins can do almost everything either natively or via its extensive library of plugins. In the same vein, always use Jenkins version control; our old build scripts used to shell out to svn.exe and run the checkouts that way, into a static folder. This meant that we couldn't run builds in parallel, and if an SVN operation were interrupted, the working copy would lock and would need to be manually cleaned up.
For complex builds, write a script that is kept in the root of the project, then after checkout, simply do ./build.sh. We used to keep all our build scripts in a separate repository which would have to be checked out onto the build node in preparation; invariably, the scripts never got updated. This way, the product and the steps required to build it are kept together, and are very unlikely to come out of sync.
Use labels as much as possible; I have our cluster set up so that builds that require dependencies are bound to label '<product>' and I add/remove the labels from the build nodes as the dependencies are added. This also means that if one node breaks and is unable to build <product>, I just remove that label and Jenkins will stop scheduling builds on it. It becomes much easier to manage than restricting builds to hostnames.
Most importantly, ensure you can always build from a fresh checkout. I cannot tell you how annoying it gets when a build fails because some tiny thing is out of alignment on a fresh build node.
I also recommend imaging your build nodes so they're quick and easy to deply, and BACK UP YOUR JENKINS CONFIG! :) When I came back from my Christmas break, it was to discover that the Jenkins server I had worked oh so hard on had been inadvertently trashed by one of the guys on my team; the disk had run out of space, so he'd remoted onto the server, gone into the Jenkins folder and deleted everything older than a month.
He hadn't thought to restrict this to the build artefacts folders. Basically lobotomised the server, and that was when we discovered we'd been backing up the build NODES, but not the MASTER itself! I was able to salvage some of the config and rebuilt the server on the latest version of Jenkins. As a result, the Jenkins folder on the master is now an SVN working copy with all the config files under version control, and a scheduled task runs daily to commit the changes.
It has been a very painful road to get this thing stable! It would be considerably easier if people weren't trying to nuke all my hard work while my back was turned...!