Planting a garden does rob the local ecosystem. Even purely native gardens lack the diversity, in particular mycodiversity, of undisturbed areas. While long term native planting can move an area back toward endemic biodiversity, it is rarely or never achieved (I'm not aware of any well-cited claims to the contrary).

It is similarly naive to view the introduction of a species through the lens of pollen quantity being 'surplus' or 'enough'. The presence of rival pollinators affects native insects whether or not there is a shortage of food - for example by discouraging presence, by occupying nesting sites, by altering endemic predation webs, and similar.