I've heard a good argument that algebra specifically should be moved the other way: teach it in elementary school, when kids are first exposed to arithmetic. There's a really natural progression from addition/subtraction/zero and multiplication/division/one to inverses and identities, and from addition/multiplication to distribution and associativity. My 3-year-old understands that if he has 5 raspberries on his fingers and he takes away one, he has 4, and then if he grabs another one and puts it on his finger, he has 5 again. Not a huge leap from there to the concept of equations, inverse operations, and solving for unknowns.