I drive my car about twice a month, never on weekdays. If someone stole my car on Monday, they could use it all week and if they returned it on Friday with a tank of gas and $0.50 per mile for wear and tear, I'd be no worse off. Yet, that would still be illegal. But I would be deprived of the opportunity to rent my car to that person for a fee of my choosing. Similarly, I'm no worse off if someone squats on land I never use, or sneaks into my theater to watch a movie when half the seats are empty. But I am deprived of the opportunity to rent that land to the squatter, or sell a ticket to the movie watcher.
Property rights don't just grant you the right not to be made worse off, they grant you the exclusive right to profit from something.
Now, whether non-violent violations of property rights should be punished by jail time is a separate issue.