More precisely, we see that galaxies started accelerating away from us a few billion years ago; before that they were decelerating (moving away from us but with the "speed" decreasing instead of increasing).

> People often point to the observation that the further away a galaxy is, the faster it appears to be moving away from us, implying acceleration.

That observation tells us that the universe is expanding, but by itself it does not tell us whether the expansion is accelerating or decelerating or neither. So you are correct that that observation alone is not sufficient to show that the expansion is accelerating.

What we look at to see how the expansion rate changes with time is a comparison of three pieces of observed data, galaxy by galaxy: redshift, brightness, and angular size. The relationship between these three quantities is what cosmologists use to construct a model of the expansion history of the universe, which in turn tells us things like what I said above, that the expansion has been accelerating for the last few billion years but before that it was decelerating.