I'm not sure that's the right line for society, but it's my personal line.
Clearly no human being is harmed by smashing the windows of a private car. (assuming that neither the smasher nor anyone else gets hurt in the actual act of the smashing.) As a first order of approximation no human got harmed, they weren't even there when their car's window got smashed. As a second order approximation obviously we know the private car's owner will need to replace the windows which will cost them.
At the same time shoplifting from Walmart is also a cost which is born by everyone who shops at Walmart. Walmart will put that cost into the price of items. They will also put the cost of anti-theft items in the price of things too. Plus those anti-theft measures will harm the non-thieves by inconveniencing them. So everyone will pay just a bit more because of the shoplifting. These costs and inconveniences add up quickly.
Why is one of these costs within the line and the other is outside the line? What makes one of them "actual human being is harmed" while the other not?
In the same way, paying hundreds of dollars to fix a broken window is a significant burden to the individual owner. Paying an extra dollar on my Walmart bill is much easier to absorb.
Both examples aren't exactly clear lines. The walmart example arguably affects "actual human beings" by raising prices, or at the very least, making the shopping experience worse. See: stores in the US where anything vaguely high value is locked up and you need to call an employee over to open it. The car window smashing example could result in no economic loss for the owner, if the owner has comprehensive 0 deductible insurance, or if the car is a company car. There's also plenty of activity that we ought to ban even if there's no "actual human being is harmed", eg. speeding, or tax evasion.
So to take a recent example from my city (one of those cities so full of rampant petty theft that stores now inconvenience everybody by locking up toiletries), you don’t have a problem with this rich asshole with a $150,000 salary and his two disrespectful, disruptive teenagers stealing steaks and Monsters from Walmart?