Also, I've never heard of muscles shrinking due to being overworked. Can you elaborate on how this happens?
For hypertrophy, past a minimal amount of stress, it increasingly seems that it doesn't matter[0].
For strength it matters.
"inclined leg press with seated row (superset 1), barbell bench press with cable hamstring curl (superset 2) and front planks (set 3) and Tuesday/Friday: machine- guided shoulder press with bicep curls (superset 1), triceps extension with wide grip pull downs (superset 2) and machine-guided knee extension (set 3). "
Both the low rep and high rep group did these supersets. It would seem like making the low rep group do supersets(extra exercises between the sets) would defeat the purpose of calling it low rep. It may have hampered the low rep groups gains.
From the link you supplied:
"Though there is no apparent advantage of lifting with different loads on changes in muscle mass, there is undoubtedly a neuromuscular advantage to lifting heavier loads if the primary outcome is performing a 1RM test (28). Conversely, it appears that periodic practice of the chosen strength outcome (e.g. 1RM) is effective at eliminating the majority of any post-training difference. "
and:
"In conjunction with previous data (28), it appears that if 1RM strength is the primary goal, performing the to-be-tested exercise with heavier loads, either consistently and/or periodically, may be required for optimal improvement."
So if you want to develop maximum strength(at the reduced rep range), you have to periodically practice in that rep range if only for the neuromuscular advantage. In this case, at least a 1RM every three weeks for the men in this study.
Stopping one rep short of exhaustion is an important technique: It doesn’t decrease your rate of gains and it increases your recovery time between sets so that you can complete all your reps (study, study, study) [3, 4].
#2: Shrinking from overworking is surprisingly a real thing. The science is unclear as to how exactly muscle mass responds to resistance exercise, so my answer would be made up if I gave you one. What I can tell you is that "being overworked" is the result of lifting too heavy for too many total reps. This might seem impossible since your body should be unable to continue lifting past a natural point of exhaustion, but if you use machine exercises (especially a pulley machine), it's easy to cause a disproportionate amount of stress to your muscle relative to the incrementing heaviness trajectory you were lifting for free weights.
[1] https://www.researchgate.net/publication/7873571_Muscular_ad...
[2] https://www.researchgate.net/publication/23252049_Changes_in...