You probably know this already, Ghostel tries to make interaction with the Emacs buffer as seamless as possible (obviously all configurable).
Of course there's line-mode where everything is just a normal buffer, like a better M-x shell replacement.
And in semi-char-mode there's a lot of config options that automatically toggle copy-mode when it makes sense, e.g. isearch, consult-line, etc. avy needs some extra config, see https://dakra.github.io/ghostel/#mouse-selection
(advice-add 'avy-action-goto :after #'ghostel-maybe-leave-input)
then you can just use avy like normal and after the jump you're automatically in copy- (or emacs-)mode and when done copying, you're automatically back to the live terminal input.