Votes are counted, not the supposed size of stage-managed "rallies".
One explanation is that voters may be enthusiastically voting "against" a particular candidate, in which case they would likely not be attending any rally or showing support in any meaningful way.
I'm not sure if this is the strongest argument in favor of your hypothesis. In most instances it would rather be an indicator of election fraud.