More and more earlier and earlier doesn't comport well with child development and can backfire by making students feel incompetent. We need better teachers, probably by paying more so more talented people join the profession.
Other countries do a lot of "pre-algebra" in the later grades of primary education, when the kids are quite ready for it; "drill and kill" rote methods are generally focused on in very early grades, since they help build familiarity with the sort of rigorous, algorithmic thinking that's required for good math proficiency. This is what Russian Math, French Math, Singapore Math, etc. are built on, and the approach has stood the test of time indeed. The fuzzy "Common Core" approach pushes abstract content way too early, and ends up confusing kids as a result.
I wholeheartedly endorse Singapore Math, Russian Math, Canadian Math (this is a thing, check out John Mighton's fantastic JUMP program). ANYTHING but US math.
I think we could pay more, but only if we could actually get better teachers. So many teachers are bad and not worth even what they get paid (personal experience from a nominally quite good school district). My concern with paying more before figuring out how to weed out bad ones is that it would just be a waste.