The smoking also produces strong odors, which (as a neighbor) I do not approve of, so I'm not sure if adding the odors from growing and processing to that will make the situation much worse?
Is there higher risk of infection transmission? I'm sure those reports a few years ago of serious lung infections from tainted vape oils are biasing my thinking here, but it also makes some sense that a high-temperature flame on a dry medium wouldn't transmit nearly as many microorganisms as a lower temperature vaporization of a wet medium.
If someone smokes a joint near your window, that smell will probably be gone in half an hour. If there is a plant that's just sitting there, it'll be more subtle, but it's not going anywhere.
That said – many people like the smell of Hopfentee (Hop tea), which is pretty similar.