Yeah, it could make sense apply it to only "non resident for tax purposes" (i.e. People filling a 1040NR)

As other comments mentioned, a tax like this should also affect rich citizens who similarly contribute to vacant houses, so to actually solve the problem I guess it should affect all property owners