The reason some pedestrians feel cyclists pose a bigger threat than drivers, is mostly due to car centric infrastructure. Cars get all the room, and cyclists&pedestrians get clumped together. Of course you see little conflicts with cars when you're physically separated from them.

Additionally it's a form of servility and motonormativity. Most people accept that they themselves have to obey cars, so no jaywalking or other behavior, most people go out of their way to avoid conflicts with cars (due to the risk), which again makes the drivers seem less of a threat. But only because you yourself have removed yourself from the situation.

In total, the numbers don't support the impression. Literally no one dies from being hit by cyclists. Vs thousands upon thousands die from being hit by vehicles each year.