It is frightening, or more like just anxiety inducing ya know. I've only had it a couple times, the worst of which was withdrawing from cigarettes. It's like you suddenly realize your experience of reality has changed. Like just now you can see the frame around your existence, like your watching through a window and not just being in it. Except that that analogy only addresses visual acuity which the full experience is that but with all senses and especially the sense of self. Derealization is really just the best word to a profoundly strange and indescribable experience.
I've also taken drugs that enforce that sensation and it's much more interesting and tolerable when you've done it intentionally, know what's happening, and that it will end. Those mind states have allowed me to explore emotions and sensations from a rather removed point of view and have some realizations (pun? Not intendes) that i wouldn't have otherwise.
If you wear glasses, chances are you've learned to ignore the constant frames around your vision. I'd like you to intentionally notice them. Do you see how the vision you experience could be likened to a movie screen? You can look around but you are trapped inside this frame. Imagine if you weren't wearing glasses but you came to notice the frame at the edge of your vision. And upon noticing, found yourself incapable of slipping back into your regular sight. Like you've been pushed back and out of yourself, and you have no way to get back in. You're still in control but it's just that, _control_. It's not you, it's just control, and you can't make it you, no matter what you do. That's at least what it's felt like for me.