Just to play devil's advocate, or perhaps help you understand how these people think, consider illegal immigration. Many people, using the exact same logic as you, consider it to be a serious issue, whereas others would say such beliefs are baseless (to put it nicely) as any facts supporting any concern are "outliers".