>What is this subject called in the US? I'd love to find out more about what that teaches and what that doesn't.

It's not a specific subject, but usually something built into the overall curriculum. It's generally called character education: here's the relevant link for CA http://www.cde.ca.gov/eo/in/se/charactered.asp

Here's the relevant link for my state (Georgia). https://www.georgiastandards.org/standards/GPS%20Support%20D...

>Wrt "not up to individual states", my understanding is that the states are responsible for the K-12 education curriculum, and so if there's moral education in K-12, I'd assume the state education board controls what goes in there. Am I wrong on this?

You are correct. However, if a state does something that clearly violates the Constitution, e.g. a state promoting a specific religion, it would eventually end up before the supreme court.