Its a matter of expectations, if you usually stay late, people will expect that you will be there. Its like if your boss gives you 10 things to do and you accomplish all of them, the next time it will give you 12 things to do, then if you stay late working on those things and you deliver the next time your boss will give you 14 things, because he is measuring your throughput of work
The solution is simple, just stand up and leave and talk to your boss about the amount of work so he can hire someone to help you or delegates the work to other person.