The problem is that many "taking something away" cases in prison can be rephrased as "not giving something."
Am I taking away your visitation rights, or am I just not giving them to you? Am I taking away your food, or am I just not giving you food?
Expectation and norms play a role in this. Suppose a judge gives a prisoner a higher sentence under the expectation that they'll work it off. They assume the prisoner will work 7 days a week, so they give a prisoner 6 years instead of 3.
Now suppose another judge gives a prisoner the correct sentence (3 years), but then says, "for every day of your original sentence that you don't work, you will spend one additional day in prison."
You're arguing that there's a fundamental difference between these two scenarios, but what is it? It's certainly not a difference for the prisoners -- they have the exact same incentive structure, and their outcomes and choices are exactly the same. It's not any different for the people benefiting from their labor; the work that they produce will look exactly the same. It doesn't even reveal any compelling differences in the judges' motivations -- both judges might reach their decision for the exact same same reasons.
So where is the difference?