https://weaviate.io/blog/cross-encoders-as-reranker
but these are significant because they noticeably improve search ranking results and a development like that happens every 5-10 years or so. Getting a bunch of results from Bing and then reranking them with a cross-encoder is a low-hanging fruit in terms of making a better than status quo search engine.
If the cross-encoder is well-calibrated then you really can merge search results from different sources the way IBM Watson did back in the day and not have to calibrate the individual sources.
1) Re-ranking is only good as the results you originally get. If you get garbage in, you will likely get garbage out, no matter how you re-rank
2) Re-renking using semantic search does little to help with the main problem in search (SEO spam, low quality content). These 'bad' pages typically rank well in semantic search and re-ranking would probably just help boost them.
You're certainly not going to run them for every document in the collection but can you run them for the top N? I think the answer to that is yes.
It takes > 1 second(perhaps 1.5-3s if you embed paragraphs not sentences) for re-ranking k >= 8 on a decent say 32GB CPU instance..(requires encoding each document passage with the search query top k times)... if you have GPUs available for your search inference sure it's just a fraction of a second, but right now I would say it's significantly more onerous than bicoders that get you the first jump in semantic search performance (10-40x slower in practise on CPUs) so the use case differs.