It's a common way of describing that shared experience. There's no need to restrict our speech or feelings to the strictly literal. If you don't like the phrase you don't have to use it, but to many couples the experience is meaningful enough that the phrase is also meaningful.