Caesar changed it to start on January 1st because the Roman consuls took office then on that day.
He also named July after himself!
It's so weird to me that they assassinated him, but didn't rename the month back. Imagine Trump had renamed June to "Trump", he then gets impeached and 2000 years later the month is still called Trump.