I personally use this (alongside an extremely high threshold):

    (add-function :after
                  after-focus-change-function
                  (lambda () (unless (frame-focus-state) (garbage-collect))))

So Emacs GCs whenever it loses focus (i.e. I'm looking at some docs in a browser or whatever). This works great for me - it's rare that I notice it still cleaning up when I switch back.