#1: Reps. When working out for muscle size (as opposed to strength), it's optimal to use a weight that’s light enough to do a set of at least 8 reps with but heavy enough that you can't easily do more than 10 reps [1, 2]. This range of 8 to 10 reps means it's fine if you stop at 8, 9, or 10 reps in any set. Go as high as you can while stopping one rep short of the maximum you feel you could do.

Stopping one rep short of exhaustion is an important technique: It doesn’t decrease your rate of gains and it increases your recovery time between sets so that you can complete all your reps (study, study, study) [3, 4].

#2: Shrinking from overworking is surprisingly a real thing. The science is unclear as to how exactly muscle mass responds to resistance exercise, so my answer would be made up if I gave you one. What I can tell you is that "being overworked" is the result of lifting too heavy for too many total reps. This might seem impossible since your body should be unable to continue lifting past a natural point of exhaustion, but if you use machine exercises (especially a pulley machine), it's easy to cause a disproportionate amount of stress to your muscle relative to the incrementing heaviness trajectory you were lifting for free weights.

[1] https://www.researchgate.net/publication/7873571_Muscular_ad...

[2] https://www.researchgate.net/publication/23252049_Changes_in...

[3] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4731492/

[4] http://www.ncbi.nlm.nih.gov/pubmed/21986694