In newer processors there exists a cache for return addresses Return Stack Buffer (RSB).
But there is a penalty for doing call and never doing ret.
From Intel's Optimization Reference Manual:[2]
"The return address stack mechanism augments the static and dynamic predictors to optimize specifically for calls and returns. It holds 16 entries, which is large enough to cover the call depth of most programs. If there is a chain of more than 16 nested calls and more than 16 returns in rapid succession, performance may degrade.
[...] To enable the use of the return stack mechanism, calls and returns must be matched in pairs. If this is done, the likelihood of exceeding the stack depth in a manner that will impact performance is very low.
This trick is also what I understand retpolines are about:
Citing kernel doc
The kernel can protect itself against consuming poisoned branch target buffer entries by using return trampolines (also known as "retpoline") for all indirect branches. Return trampolines trap speculative execution paths to prevent jumping to gadget code during speculative execution. x86 CPUs with Enhanced Indirect Branch Restricted Speculation (Enhanced IBRS) available in hardware should use the feature to mitigate Spectre variant 2 instead of retpoline. Enhanced IBRS is more efficient than retpoline.
retbleed https://lwn.net/Articles/901834/
[2] https://discourse.llvm.org/t/is-pic-code-defeating-the-branc...