I detailed a bit more in my other rant, but if you want to stay or get what you're owed, you're going to have to risk the conflict.

The only way to not have conflict about this is to leave and say nothing. If you think he won't allow you to leave at all without conflict, then you might as well choose the conflict with the better possible rewards.

You shouldn't let things continue. But whether or not you're going to attempt to point out the problem is up to you.