In practice, a lot of relevancy tuning is about properly factoring in other "signals", such as something's recency, distance, popularity, stock availability, the section of the website being searched from, etc.
Thus, there's often a lot more than just the textual relevancy being involved.
Postgres will never compromise on MVCC correctness, and therefore doesn't do any caching of partial results beyond having things in shared buffers and/or the page cache.
Elasticsearch will err on prioritising speed and cacheability. The "signals" that appear again and again in the same queries will likely be pretty compactly cached as (roaring) bitmaps in a filter cache. That enables potentially being a _lot_ faster, because it's ok with "you're seeing approximately everything that was ingested about a second a go".
In addition to that, Elasticsearch(/Lucene) comes with a lot more tooling for text processing, stemming, compound word splitting, etc.