How big is the gap? I did some testing & found that so long as it's similar domain (eg don't pre-train for COCO on X-Rays) it still helps. https://blog.roboflow.com/transfer-learning-similarity/

The intuition for this is that lines, colors, textures, shapes are all general concepts that can be learned from a different domain & used in the earlier layers of the model to build up to more complex features.

Coming to the conclusion that models don't generalize well could be because you are seeking to create solutions to more interesting problems. I don't spend resources on an AI team and have been surprised as how well models do generalize.