You're definitely not a weirdo edge case, I think we all feel something like that around email, facebook notifications etc, and it's something we very consciously want to avoid in our product.
Each Daily Calm session is available for that day only, so you either listen to it or you don't. If you do come back to the app every day there's something fresh for you if that's what you'd like (you can of course to a regular session too). If you don't use the app on any particular day, no big deal, there's no backlog of content for you to stress about catching up with!