This is roughly what the Palestinians did when they found themselves attacked by the Israeli settlers, back at the formation of Israel. It's important to always remember that in this conflict, Israel has been the first aggressor, not the defense. And since Israel had the full support of the UN, and later of the USA, the Palestinians really couldn't spend too much time waiting for those to fix their problem (though they are actively engaged even today).
Now, for Israel to stop these attacks on their own people, especially before October 7th, the solution is actually relatively simple: stop attacking the people of Palestine, and they will stop attacking back. Recognize Palestine as a state, including a corridor to link Gaza and the West Bank, give back the territory stolen by illegal colonists in the West Bank, stop blockading their borders. Condition this on a free cross-Palestinian election, maybe even explicitly blocking Hamas from it.
The problem is that Israel's goal is not simply to stop the attacks on their civilians. Israel's goal, as many top leaders in Israel, including Netanyahu, openly declare, is to expand Israel's borders while excluding Palestinians from being Israeli citizens. Netanyahu has been supporting Hamas for more than a decade, and has many times openly admitted to this, as a great ally in ensuring that a two-state solution will never be reached.
So, what if a few Israeli citizens are killed every year? That's a price that Israel's government is willing to pay to keep the status quo and not get their hands too dirty with Palestinian blood. Of course, Oct 7th changed that: it challenged them and showed they are not all powerful, and that is not acceptable, so the people of Gaza had to be punished.
And just one final note - Israel performed two October 7th-level attacks just in the last twenty years - 2004's Operation Cast Led (1200+ civilians killed), and 2014's Operation Protective Edge (2200+ civilians killed). Except of course those were much worse, since Israeli bombs also destroyed many thousands of homes, displacing tens of thousands of civilians atop those killed.