Because a lot of it _is_ spiritual mumbo-jumbo.
The books "The Mind Illuminated" and "Mastering the Core Teachings of the Buddha" occasionally get recommended on here and are pretty mainstream books, supposedly backed by "brain science" and written by two PhDs. Let me give an excerpt from "The Mind Illuminated":
"The first practice involves cultivating the so-called “higher knowledges of the mundane type.” These are: 1. The “higher powers,” which are said to allow a yogi to perform miracles such as walking on water, or walking through walls. [...] 4. Knowing the minds of others, which is a form of telepathy. 5. Recollecting past lives"
And this isn't presented as a "oh, here's this historical context", it's presented without any real comment next to the jhanas.
Or, a nice bit in "Mastering the Core Teachings of the Buddha", talking about pyschic powers:
"On the other hand, it does seem to be possible through powerful intent, strong concentration ability, appreciation of interdependence and careful experimentation to manipulate what we might call “this world”,as well as those in it, in very unusual and profound ways. Yes, I am referring to such things as telekinesis, mind control, reading other peoples thoughts, pyromancy, and all of that. The more you get your concentration and insight trips together and the more you look into the magical aspect of things, the more you will learn about what I will call the magical laws of the universe and how to use your will to manipulate it."
You know why people are skeptical of things like "awakening"? Because it's sold in the same breath as all the religious parts, making it impossible to discern which is what. If the cost of getting to 'the deep parts' means having to not dismiss such obvious bullshit like pyromancy, I think society will do fine with such 'disservice'.