You live in a society that has some really strong stereotypes of early risers and night owls so there is a large amount of learned values involved here. There might be more to it than that for you, but the learned shame part is real and a very consistent part of our culture.