Most "scientific studies" in this area are worthless as they use untrained college age males and only follow them for 1-3 months. Virtually any sort of training will produce significant gains in these subjects and the variance will usually be due more to outside factors than the training regimen itself.
As for the "why", this is obvious to anyone who actually trains with heavy weights. Perform 8 reps of an exercise to failure. Rest 30 seconds. Repeat. I guarantee you will not be able to perform anywhere near 8 reps on the second set, you might not even be able to perform 4. OTOH, rest 4 minutes and you very likely will be able to perform 7 if not 8 reps.
Again, I totally get your point about broscience vs actual science, but this is one of those instances where I'm not sure the "30 seconds between sets" even rises to the level of broscience. You read about it in meathead magazines and some may even claim to adhere to it to demonstrate how manly they are, but if you go to the gym and watch serious lifters lifting heavy weights, you will not find them only resting 30 seconds between sets.