As another poster linked, you may be a slow caffeine metabolizer. I am and avoiding caffeine entirely makes a big difference.

Another thing that might be making a difference is what you do in the hour to two hours before sleep. If I do anything exciting, like sports, suspenseful media & games, or anything analytical, then that will delay how quickly my body relaxes into later in the night, which messes up the beginning of the night when deep sleep mostly occurs.

Lastly, if you're measuring your deep sleep based off of a device that isn't on your head, then take that data with a huge grain of salt. I compared sleep data from an Oura ring with the Dreem 2 headband and the ring was consistently so wrong as to be useless for driving better sleep behavior.