It definitely needs to be accounted for. In the past decade, there has been a proliferation of smaller marathons. If you made a plot of "100th finisher times" by # of participants, it would be slower for smaller races. If the growth in marathons is primarily on the smaller end, then you'd expect the average to go down.