Sorry, what's your source? Your link fails to address your assertion at all.
Any website key retention is out of convention rather to ensure data access, and a convention Let's Encrypt is fast upending:
The certificate is valid for 90 days, during which renewal can take place at any time.
https://en.wikipedia.org/wiki/Let%27s_Encrypt
Key exchange is a component of PKI. You either need to know, or be able to request, a given key to validate it, or transmissions based on it.
TLS (as SSL before it) relies on certificate authorities (CAs), whose keys 1) are distributed to all major browsers and/or operating systems, and 2) which then vouch for website keys. Other mechanisms (e.g., key pinning) are also used. This model has many critics.
PGP (or Gnupg, and alternate implementation) rely, also to various degrees of general criticism, on a web of trust model, in which keys are signed by various other keys, possibly but not necessarily the user's own. Trust is ideally personally verified. It's been observed that this poses scaling issues.
Other PKI models have their own approaches to key trust. S/MIME uses CAs similarly to TLS/SSL, for key authentication, but keys themselves must still be retained, and secured, to assure future data access and integrity. SSH effectively relies on TOFU (trust on first use) or OOB (out of band) key distribution -- both of which effectively punt the process.
TLS, SSL, and SSH are all session-based (data-in-flight) protocols. PGP and S/MIME are storage-oriented, data-at-rest protocols. My initial comments apply to each.