I pay for YouTube Premium, do the steps you describe, and do not see that behavior.
You must be in another browser session that's not signed in to your Premium account. You're in incognito or on second browser. Just sign in to YouTube to fix it.
For anyone else reading: the issue happened on iOS Safari. I was signed in on the YouTube iOS app, and I was signed in on Google.com on Safari, so I assumed there wasn't anywhere else I needed to be signed in. But sure enough, I went to youtube.com in Safari and wasn't signed in. Tried it once after doing that and it worked. Will see if it solves it, seems likely.