I'm sure you already know, but it's completely irrational to fear the things out of your control. There's literally nothing to be accomplished by doing so.
Fear is the foreknowledge of (or of the risk of) something unpleasant happening to you. Foreknowledge of our inevitable death, over which we have no control (sure, we can try to prolong life a bit, or end it early, but the ultimate result is the same), induces fear, ranging from anxiety to terror, in large numbers of people - perhaps even the majority.
To me, it's irrational to fear events that are highly improbable, but in view of our survival instinct as humans, it's quite rational to fear something horrible that is absolutely going to happen to you, and there's nothing you can do about it - for example, being scheduled for judicial execution the next day.
B: People are not perfectly rational.
However, I totally agree with your main point.
For example, on a policy level, Japan never has them. They just don't immigrate any of the religious group who might commit terrorist attacks. So it's a non-issue.
And on a personal level, you can mitigate terror attack risks. Just avoid gatherings of people, basically. You can go as far as you want with this - living off-grid in the forest makes you pretty much fully insulated from terrorist attacks.
So the only question remaining on a personal and policy level is what the cost/benefit is for various things you could do to prevent terrorist attacks.
This is a common mistake. People like to think that some options are impossible, or some gains are free. Thinking this way makes decisions easier; it makes you feel more secure in your opinions. In reality, everything is possible, and everything costs.
This is regularly triggered by various disputes about commonly claimed territory and things such as politicians visiting shrines that house the Japanese war dead (including war criminals). It often leads to rioting and destruction of perceived Japanese things (cars, restaurants, etc) in China.
Still no terrorist attacks in Japan committed by Chinese though despite Japan allowing in large numbers of Chinese visitors, students and workers.
I have a Japanese friend that has been road-tripping Europe for some time in his vacations and has been unable to convince any (Japanese) friend to go with him. In the last one, he was almost coming with a guy, but at the last moment, that guys mother convinced him to cancel the trip, and this was before the Paris attacks.
I remember being a kid living in Japan, and watching the subway attacks on TV. It was scary.
But to your point, I agree, statistically, Japan is about the safest country there is in terms of human on human violence.
cough