How do you handle the I and D term then?
What if the player can also use a gamepad and go from left to right gradually?
The fundamental problem is that the error accumulates while you try to reach the target. But then the target suddenly changes. Do you just drop the previous error? Do you create an upper bound on the error? Or something else? This is the question no PID post ever answers.
I don't say, that this is impossible, but just that PID's in this context are harder to use than they first appear.