That seems oddly narrow. As a contrived example (I like stress-testing ethical intuitions) would you consider barricading the doors and windows of someone's home and waiting for them to starve to death to qualify as physical violence? (Assume for the sake of argument that noone else interferes with this, perhaps because you're in a isolated rural area.)
(I'll also perfunctorially object to terms like "man" or "human", but I doubt that's a contextually relevant distinction, given linguistic history and the lack of any other sapient species respectively.)