I am not an expert on the subject but IMO Taoism has a very similar phenomenon to Buddhism (and indeed all religions or popular philosophies really) in that the “classical”, original, core teachings are pretty different from the organized later movements under the same name. So the context - philosophical, historical, sociological, spiritual - in which you approach the subject and whether you’re doing so on the basis of the original thing or its more organized movements might make it so people interested in it under different contexts both think the other is ignorant or incorrect.
I haven’t read the Tao of Pooh myself but I want to point this out because I think it’s possible to understand Taoism (the classical philosophy) and concepts like Wu-wei without necessarily knowing anything about Neidan and or “Taoist Magic”