It's not realistic to expect patents to be granted more intelligently. We are by definition, in the hypothetical best case (all honest actors), talking about the absolute bleeding edge of technology, where you are going to need real experts to be able to discern what is "real innovation" and what isn't. I don't even consider myself capable of analyzing patents except for in an incredibly narrow slice of computer science where I've worked for 10 years, so I'm not sure where you expect to find these generalists who will one day be looking at AI algorithms and the next day be judging graphics code. The only alternative would be to literally have an expert from every subfield. Even if you had infinite funds to fill this magical patent office, why would such experts take this boring job (unless their goal is to work on physics on the side ;) )? As it stand today btw, most patent clerks being hired are engineers straight out of college, and I imagine they aren't exactly the top of their class. No one dreams of being a patent clerk.
This is all exacerbated by the fact that the patent agency is a government agency, where there really isn't anything incentivizing them to get better. And yet on the other side you have lawyers who are 100% incentivized to create the most convoluted patents to actively confuse (the already ill-equipped) patent clerks and juries.
I also agree with the statement "if we could just grant the right patents then this wouldn't be a problem". I just don't see any reasonable way to increase the fidelity of patent grants, nor a reasonable fallback system for when a bad patent DOES inevitably get granted (trial by jury for discerning innovation is probably the worst method).
1000 years ago you could be a mathematician, poet, and biologist. Today, you have to go to school for 5 years to get a PhD in an incredibly focused subfield. 100 years ago the patents you were dealing with were "machine that makes light". Now its methods of improving graphics performance when running on this super specific kind of computer architecture. It might have been reasonable for an outsider to make a judgement call before, but I seriously don't think it is anymore.