A patent owner would "lose" a patent only if someone were to successfully challenge the patent's validity or enforceability, either in court or in a USPTO proceeding. If the patent owner were to fail to defend against the challenge, then yeah, the court or USPTO might invalidate the patent.

Now to be sure, if a patent owner were to wait too long to assert a patent against an infringer, that could complicate the patent owner's efforts to get royalties or an injunction (a stop-use order) later on against that infringer, and possibly against other infringers.

And any infringer might try to use the patent owner's delay as proxy evidence that the patent owner didn't really believe that the patent was valid.

But failure to assert a patent, in and of itself, will normally have no bearing on a patent's validity or life.