There is a difference between non-native and invasive species.
Non-native species: These are species that have been moved from their native habitats to a new environment, usually as a result of human activities. Non-native species can be any type of organism, including plants, animals, fungi, and bacteria.
Invasive species: These are non-native species that not only survive in their new environments but also reproduce rapidly and spread widely, causing harm to the local ecosystem, economy, or human health. They can out-compete native species for resources, alter habitats, and disrupt ecosystem functions.
Here is a study on Stinknet in the Sonoran Desert. If you read through it I think most would say is qualifies as invasive in this environment https://sdcwma.org/docs/stinknet_desert_plants.pdf
Conservation isn't about keeping the environment in a bubble but rather being stewards promoting the overall health of the ecosystem.