Now if I step into a plane with passengers (even if it's the same small Cessna), suddenly I'm no longer allowed to fly single pilot IFR operations without autopilot, need much higher fuel reserves, need to plan an alternate, check weather requirements and have done a 6-monthly checkflight and training. And for most airplanes I would also need a co-pilot, so there are two people monitoring everything.
Most small airplane crashes are not due to engine or equipment failure, mostly it's pilot error, and that's where all the extra procedures in commercial aviation really help.
Well maintained single engine aircraft (in the certified category) are generally reliable. The majority of incidents in these planes come down to avoidable pilot error. Looking at the forces behind pilots making mistakes will provide a lot of insight. General aviation is not forgiving to mistakes and tired, under-trained, overconfident or briefly otherwise human behavior can (and does routinely) cause issues. These factors are more carefully compensated for on the commercial side.
Of course, the counter argument here is the missing safety features that should exist to compensate for mistakes.
A lot of GA -- certainly anything under around $200k -- is a fair-weather activity. Planes can't really fly in the winter -- ice builds up on wings. Strong winds throw small airplanes around. There isn't instrumentation to land with limited visibility from fog, severe snow, or severe rain. Etc.
Weather can change unpredictably. If you've flown for a weekend getaway Friday, the weather turns over the weekend, and you have an important business meeting Monday morning, do you risk flying back? Do you miss your meetings? Do you leave your airplane there and fly back commercial, with thousands of dollars in hanger+rental fees and days of hassle to get your airplane back?
It's a value call. Different people have different risk tolerances based on personality, life situation, and culture.
A lot of this applies to cars too actually -- if everyone drove conservatively, they'd be pretty safe. At least where I live, virtually everyone speeds. People cut other cars off. Etc.