I wouldn‘t stress the fact it‘s a stray cat too much. In my experience the situation wouldn‘t be any different if the aggressive cat had an owner in the neighborhood.
Disclaimer: I don‘t have cats myself but all the cats in my area (non stray) seem to use my ground as their toilette :(
That’s because you don’t have a cat. You can have a cat for him to defend your ground. Or, much better, a dog.
The stray presumably can’t unlock and open house doors, which would imply theirs are outside at times.