If the rising pollen counts are part of a process that is driven by the overall global temperature trend but which is relatively insensitive to annual variation around that trend, such that the output is smoothed compared to the input, that would produce the pollen counts rising every year with a rising global temperature trend, even if global temperatures don't rise every year.
One particular reason this might be the case is if the process driving the increased pollen count is a reaction to global temperature (or some other more direct product of global temperature) that moves to slowly compared to the average rate of global temperature change to ever reach an equilibrium point with the current global temperature, so that its always trailing well below what it would be at the current global temperature if that temperature were a long-term sustained average.