This gives rise to problems for computers since they are irregular and unpredictable, and can't be made part of algorithms. In extent, to properly calculate a precise time difference (actual elapsed time) over several years, you actually need to consult a table of added, historic leap seconds and take these into account too...
So the day is not 24 hours or 86400 seconds, but rather 86400.002 seconds on average. That's about 2 milliseconds or more of deviation per day and for a whole year that's about 0.7 or 0.9 secs worth of deviation. Yet we still pretend that the day has precisely 24 hours, hence the need for leap seconds.
Given that since 40 years ago since leap seconds were adopted about 25 leap seconds have been scheduled, that sounds about right.
Cumulative deviation since 1972 will be 26 seconds.
26 seconds over 43 years (1972-2015) is 0.60 seconds/year. If this matches the average offset from 1820...1972 (which would add up to about 92 seconds), it means that our average is wrong by ~0.6 seconds, but the time length doesn't gain additional seconds due to "spinning down" (caused by tidal interaction with the moon, or whatever other effects there might be).
That's actually also what the graph suggests: It's havnig a short-term daviation of +/- 1ms over the course of (judging by eye) weeks, and a long-term deviation of +/- 2ms over the course of decades. The average deviation of 0.6seconds/year (number of leap seconds inserted) translates to an average of 1.65ms/day from 1972-now.
Much of the slowing is caused by tidal friction with the Moon (which recedes slightly each year), iirc eventually the Earth will become tidally locked with the Sun (same side facing all the time).
In practice, it would take so long for the Earth to tidal-lock itself to the Moon, it's most likely some other major event will happen first (like the Sun expanding and swallowing the Earth).