I would say the trick is making the problem statement coherent. Sure, coming up with a firearm that can't kill innocent people is impossible because the innocence of someone is subjective. So instead, you can choose to approximate a solution and then iterate until you have a solution that is technically imperfect but so close to your initial goal that the difference is negligible.
For instance, in your firearm example, we can't determine who is innocent and who isn't. But we can use embed a facial recognition device into the firearm that then cross references whoever the firearm is pointed at against a database of known non-innocents.
And then exclude any targets obviously under 18.
And then make the model probabilistic and add greater weight to targets that appear to be carrying weapons of their own or that are acting in an obviously dangerous manner.
You get the point. After enough time, you would have a firearm that is (arguably) better off than the one you started off with and so closely approximates the goal of your initial problem description that you don't care to make a distinction.
I could be wrong, but I imagine this is how AGI safety researchers think about their work to some degree.