I'm reading a story on a page with a back to the top button. I get distracted in real life and change the grip on my phone. I look back at my phone to continue reading and I'm at the top of the page. Now I have to find where I was again.
I've probably lost more time from accidentally hitting the back to the top button than the button has saved me time. This has made me paranoid about those buttons and I definitely prefer sites not to have them. Sometimes I switch to desktop mode on my phone just so I don't have that button on the screen.